2024-01-24T13:15:01.093 INFO:root:teuthology version: 0.0.1.dev251+gc856e0f 2024-01-24T13:15:01.093 DEBUG:teuthology.run:Teuthology command: teuthology --name vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi --owner scheduled_vshankar@teuthology --description fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}} --archive /home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530810 --verbose -- /home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530810/orig.config.yaml 2024-01-24T13:15:01.145 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-24T13:15:01.227 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530810 branch: wip-vshankar-testing1-reef-2024-01-24-1150 description: fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}} email: vshankar@redhat.com first_in_suite: false job_id: '7530810' kernel: client: sha1: distro flavor: default kdb: true sha1: f2d3460408951339ac0933340ad449487c9efddc ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-vshankar-testing1-reef-2024-01-24-1150 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - slow metadata IO - SLOW_OPS - slow request sha1: 4e3182a65800aed6d4aa2f6b86da3cfd72188134 subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported dry_run: true install: ceph: flavor: default sha1: 4e3182a65800aed6d4aa2f6b86da3cfd72188134 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - nowsync syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing1-reef-2024-01-24-1150 sha1: 4e3182a65800aed6d4aa2f6b86da3cfd72188134 owner: scheduled_vshankar@teuthology priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 4668 sha1: 4e3182a65800aed6d4aa2f6b86da3cfd72188134 sleep_before_teardown: 0 subset: 346/1024 suite: fs suite_branch: wip-vshankar-testing1-reef-2024-01-24-1150 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_4e3182a65800aed6d4aa2f6b86da3cfd72188134/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 4e3182a65800aed6d4aa2f6b86da3cfd72188134 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi029.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCiK2VIXd6Nu30jJMgSMRgo2UZyHyGkunsspZGUXTboTnLdPBVmazxowZHf2sjurpFHT6+0u0dIu7055V2p51ig= smithi163.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGzEy4v5oV9PYT1rSTFgvNHNhlEnLzAgJW98x7PDsgydywKRd7tBCIF7SJdjd9v4Xsc7+BZF15W50BT3NTeJsKw= smithi187.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCXDV7vpbfuct6+Hq8JqpQryO10457dw7LMPhk4rLbFIUjKm0rpSLKVUdEk5m5bb4ZdqV+vw4Q3gaRW/IQuKL3o= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/iozone.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: c856e0f9244dd29a50697123b480d63f16a4ed81 timestamp: 2024-01-24_12:12:30 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-01-24T13:15:01.227 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_4e3182a65800aed6d4aa2f6b86da3cfd72188134/qa; will attempt to use it 2024-01-24T13:15:01.228 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_4e3182a65800aed6d4aa2f6b86da3cfd72188134/qa/tasks 2024-01-24T13:15:01.228 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-24T13:15:01.230 INFO:teuthology.task.internal:Checking packages... 2024-01-24T13:15:01.252 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '4e3182a65800aed6d4aa2f6b86da3cfd72188134' 2024-01-24T13:15:01.252 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-24T13:15:01.252 INFO:teuthology.packaging:ref: None 2024-01-24T13:15:01.252 INFO:teuthology.packaging:tag: None 2024-01-24T13:15:01.252 INFO:teuthology.packaging:branch: wip-vshankar-testing1-reef-2024-01-24-1150 2024-01-24T13:15:01.252 INFO:teuthology.packaging:sha1: 4e3182a65800aed6d4aa2f6b86da3cfd72188134 2024-01-24T13:15:01.253 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-vshankar-testing1-reef-2024-01-24-1150 2024-01-24T13:15:01.469 INFO:teuthology.task.internal:Found packages for ceph version 18.2.1-341.g4e3182a6 2024-01-24T13:15:01.471 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-24T13:15:01.478 INFO:teuthology.task.internal:no buildpackages task found 2024-01-24T13:15:01.478 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-24T13:15:01.485 INFO:teuthology.task.internal:Saving configuration 2024-01-24T13:15:01.504 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-24T13:15:01.510 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-24T13:15:01.533 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530810', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-24 13:09:09.630991', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCiK2VIXd6Nu30jJMgSMRgo2UZyHyGkunsspZGUXTboTnLdPBVmazxowZHf2sjurpFHT6+0u0dIu7055V2p51ig='} 2024-01-24T13:15:01.549 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi163.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530810', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-24 13:09:09.629150', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGzEy4v5oV9PYT1rSTFgvNHNhlEnLzAgJW98x7PDsgydywKRd7tBCIF7SJdjd9v4Xsc7+BZF15W50BT3NTeJsKw='} 2024-01-24T13:15:01.570 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi187.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530810', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-24 13:09:09.632676', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCXDV7vpbfuct6+Hq8JqpQryO10457dw7LMPhk4rLbFIUjKm0rpSLKVUdEk5m5bb4ZdqV+vw4Q3gaRW/IQuKL3o='} 2024-01-24T13:15:01.570 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-24T13:15:01.577 INFO:teuthology.task.internal:roles: ubuntu@smithi029.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-01-24T13:15:01.577 INFO:teuthology.task.internal:roles: ubuntu@smithi163.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-01-24T13:15:01.577 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-01-24T13:15:01.577 INFO:teuthology.run_tasks:Running task console_log... 2024-01-24T13:15:01.674 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff4fc7bbf70>, signals=[15]) 2024-01-24T13:15:01.674 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-24T13:15:01.681 INFO:teuthology.task.internal:Opening connections... 2024-01-24T13:15:01.681 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi029.front.sepia.ceph.com 2024-01-24T13:15:01.683 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-24T13:15:01.757 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi163.front.sepia.ceph.com 2024-01-24T13:15:01.758 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-24T13:15:01.827 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi187.front.sepia.ceph.com 2024-01-24T13:15:01.828 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-24T13:15:01.898 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-24T13:15:01.905 DEBUG:teuthology.orchestra.run.smithi029:> uname -m 2024-01-24T13:15:01.927 INFO:teuthology.orchestra.run.smithi029.stdout:x86_64 2024-01-24T13:15:01.927 DEBUG:teuthology.orchestra.run.smithi029:> cat /etc/os-release 2024-01-24T13:15:01.988 INFO:teuthology.orchestra.run.smithi029.stdout:NAME="Red Hat Enterprise Linux" 2024-01-24T13:15:01.988 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION="8.6 (Ootpa)" 2024-01-24T13:15:01.988 INFO:teuthology.orchestra.run.smithi029.stdout:ID="rhel" 2024-01-24T13:15:01.989 INFO:teuthology.orchestra.run.smithi029.stdout:ID_LIKE="fedora" 2024-01-24T13:15:01.989 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_ID="8.6" 2024-01-24T13:15:01.989 INFO:teuthology.orchestra.run.smithi029.stdout:PLATFORM_ID="platform:el8" 2024-01-24T13:15:01.989 INFO:teuthology.orchestra.run.smithi029.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-01-24T13:15:01.989 INFO:teuthology.orchestra.run.smithi029.stdout:ANSI_COLOR="0;31" 2024-01-24T13:15:01.989 INFO:teuthology.orchestra.run.smithi029.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-01-24T13:15:01.989 INFO:teuthology.orchestra.run.smithi029.stdout:HOME_URL="https://www.redhat.com/" 2024-01-24T13:15:01.989 INFO:teuthology.orchestra.run.smithi029.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-01-24T13:15:01.989 INFO:teuthology.orchestra.run.smithi029.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-24T13:15:01.989 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:15:01.989 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-24T13:15:01.989 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-01-24T13:15:01.989 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-01-24T13:15:01.989 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-01-24T13:15:01.990 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server 2024-01-24T13:15:02.014 DEBUG:teuthology.orchestra.run.smithi163:> uname -m 2024-01-24T13:15:02.034 INFO:teuthology.orchestra.run.smithi163.stdout:x86_64 2024-01-24T13:15:02.035 DEBUG:teuthology.orchestra.run.smithi163:> cat /etc/os-release 2024-01-24T13:15:02.093 INFO:teuthology.orchestra.run.smithi163.stdout:NAME="Red Hat Enterprise Linux" 2024-01-24T13:15:02.094 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION="8.6 (Ootpa)" 2024-01-24T13:15:02.094 INFO:teuthology.orchestra.run.smithi163.stdout:ID="rhel" 2024-01-24T13:15:02.094 INFO:teuthology.orchestra.run.smithi163.stdout:ID_LIKE="fedora" 2024-01-24T13:15:02.094 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION_ID="8.6" 2024-01-24T13:15:02.094 INFO:teuthology.orchestra.run.smithi163.stdout:PLATFORM_ID="platform:el8" 2024-01-24T13:15:02.094 INFO:teuthology.orchestra.run.smithi163.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-01-24T13:15:02.094 INFO:teuthology.orchestra.run.smithi163.stdout:ANSI_COLOR="0;31" 2024-01-24T13:15:02.094 INFO:teuthology.orchestra.run.smithi163.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-01-24T13:15:02.094 INFO:teuthology.orchestra.run.smithi163.stdout:HOME_URL="https://www.redhat.com/" 2024-01-24T13:15:02.094 INFO:teuthology.orchestra.run.smithi163.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-01-24T13:15:02.094 INFO:teuthology.orchestra.run.smithi163.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-24T13:15:02.094 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-01-24T13:15:02.094 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-24T13:15:02.094 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-01-24T13:15:02.095 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-01-24T13:15:02.095 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-01-24T13:15:02.095 INFO:teuthology.lock.ops:Updating smithi163.front.sepia.ceph.com on lock server 2024-01-24T13:15:02.114 DEBUG:teuthology.orchestra.run.smithi187:> uname -m 2024-01-24T13:15:02.134 INFO:teuthology.orchestra.run.smithi187.stdout:x86_64 2024-01-24T13:15:02.135 DEBUG:teuthology.orchestra.run.smithi187:> cat /etc/os-release 2024-01-24T13:15:02.194 INFO:teuthology.orchestra.run.smithi187.stdout:NAME="Red Hat Enterprise Linux" 2024-01-24T13:15:02.194 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION="8.6 (Ootpa)" 2024-01-24T13:15:02.194 INFO:teuthology.orchestra.run.smithi187.stdout:ID="rhel" 2024-01-24T13:15:02.194 INFO:teuthology.orchestra.run.smithi187.stdout:ID_LIKE="fedora" 2024-01-24T13:15:02.194 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION_ID="8.6" 2024-01-24T13:15:02.194 INFO:teuthology.orchestra.run.smithi187.stdout:PLATFORM_ID="platform:el8" 2024-01-24T13:15:02.194 INFO:teuthology.orchestra.run.smithi187.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-01-24T13:15:02.194 INFO:teuthology.orchestra.run.smithi187.stdout:ANSI_COLOR="0;31" 2024-01-24T13:15:02.194 INFO:teuthology.orchestra.run.smithi187.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-01-24T13:15:02.194 INFO:teuthology.orchestra.run.smithi187.stdout:HOME_URL="https://www.redhat.com/" 2024-01-24T13:15:02.195 INFO:teuthology.orchestra.run.smithi187.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-01-24T13:15:02.195 INFO:teuthology.orchestra.run.smithi187.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-24T13:15:02.195 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-01-24T13:15:02.195 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-24T13:15:02.195 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-01-24T13:15:02.195 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-01-24T13:15:02.195 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-01-24T13:15:02.195 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server 2024-01-24T13:15:02.214 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-24T13:15:02.225 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-24T13:15:02.231 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-24T13:15:02.231 DEBUG:teuthology.orchestra.run.smithi029:> test '!' -e /home/ubuntu/cephtest 2024-01-24T13:15:02.233 DEBUG:teuthology.orchestra.run.smithi163:> test '!' -e /home/ubuntu/cephtest 2024-01-24T13:15:02.235 DEBUG:teuthology.orchestra.run.smithi187:> test '!' -e /home/ubuntu/cephtest 2024-01-24T13:15:02.254 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-24T13:15:02.260 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-24T13:15:02.261 DEBUG:teuthology.orchestra.run.smithi029:> test -z $(ls -A /var/lib/ceph) 2024-01-24T13:15:02.294 DEBUG:teuthology.orchestra.run.smithi163:> test -z $(ls -A /var/lib/ceph) 2024-01-24T13:15:02.296 DEBUG:teuthology.orchestra.run.smithi187:> test -z $(ls -A /var/lib/ceph) 2024-01-24T13:15:02.321 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-24T13:15:02.395 INFO:teuthology.run_tasks:Running task kernel... 2024-01-24T13:15:02.410 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': 'f2d3460408951339ac0933340ad449487c9efddc'} 2024-01-24T13:15:02.410 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-01-24T13:15:02.410 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-01-24T13:15:02.410 DEBUG:teuthology.orchestra.run.smithi029:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-24T13:15:02.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T13:15:02.430 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2024-01-24T13:15:02.491 INFO:teuthology.orchestra.run.smithi029.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-01-24T13:15:02.492 INFO:teuthology.task.kernel:Running kernel on smithi029: 4.18.0-513.9.1.el8_9.x86_64 2024-01-24T13:15:02.492 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum install -y kernel 2024-01-24T13:15:04.609 INFO:teuthology.orchestra.run.smithi029.stdout:Updating Subscription Management repositories. 2024-01-24T13:15:04.609 INFO:teuthology.orchestra.run.smithi029.stdout:Unable to read consumer identity 2024-01-24T13:15:04.873 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:15:04.873 INFO:teuthology.orchestra.run.smithi029.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-24T13:15:04.873 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:15:08.211 INFO:teuthology.orchestra.run.smithi029.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2024-01-24T13:15:12.166 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 4.1 MB/s | 16 MB 00:03 2024-01-24T13:15:16.173 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 405 kB/s | 24 kB 00:00 2024-01-24T13:15:16.986 INFO:teuthology.orchestra.run.smithi029.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-01-24T13:15:16.986 INFO:teuthology.orchestra.run.smithi029.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-01-24T13:15:17.099 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-24T13:15:17.101 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-01-24T13:15:17.101 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-24T13:15:17.263 DEBUG:teuthology.orchestra.run.smithi029:> echo no | sudo yum reinstall kernel || true 2024-01-24T13:15:17.651 INFO:teuthology.orchestra.run.smithi029.stdout:Updating Subscription Management repositories. 2024-01-24T13:15:17.651 INFO:teuthology.orchestra.run.smithi029.stdout:Unable to read consumer identity 2024-01-24T13:15:17.875 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:15:17.875 INFO:teuthology.orchestra.run.smithi029.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-24T13:15:17.875 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:15:18.244 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:00:02 ago on Wed 24 Jan 2024 01:15:16 PM UTC. 2024-01-24T13:15:18.333 INFO:teuthology.orchestra.run.smithi029.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-01-24T13:15:18.333 INFO:teuthology.orchestra.run.smithi029.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-01-24T13:15:18.394 INFO:teuthology.orchestra.run.smithi029.stderr:Error: No packages marked for reinstall. 2024-01-24T13:15:18.450 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum reinstall -y kernel || true 2024-01-24T13:15:18.835 INFO:teuthology.orchestra.run.smithi029.stdout:Updating Subscription Management repositories. 2024-01-24T13:15:18.835 INFO:teuthology.orchestra.run.smithi029.stdout:Unable to read consumer identity 2024-01-24T13:15:19.069 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:15:19.069 INFO:teuthology.orchestra.run.smithi029.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-24T13:15:19.069 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:15:19.432 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:00:03 ago on Wed 24 Jan 2024 01:15:16 PM UTC. 2024-01-24T13:15:19.504 INFO:teuthology.orchestra.run.smithi029.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-01-24T13:15:19.504 INFO:teuthology.orchestra.run.smithi029.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-01-24T13:15:19.541 INFO:teuthology.orchestra.run.smithi029.stderr:Error: No packages marked for reinstall. 2024-01-24T13:15:19.594 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q kernel | sort -rV | head -n 1 2024-01-24T13:15:19.766 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-01-24T13:15:19.766 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-01-24T13:15:19.766 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-01-24T13:15:19.766 DEBUG:teuthology.parallel:result is None 2024-01-24T13:15:19.767 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-24T13:15:19.773 INFO:teuthology.task.internal:Creating test directory... 2024-01-24T13:15:19.774 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-24T13:15:19.776 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-24T13:15:19.778 DEBUG:teuthology.orchestra.run.smithi187:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-24T13:15:19.798 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-24T13:15:19.805 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-24T13:15:19.812 INFO:teuthology.task.internal:Creating archive directory... 2024-01-24T13:15:19.812 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-24T13:15:19.835 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-24T13:15:19.839 DEBUG:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-24T13:15:19.877 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-24T13:15:19.883 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-24T13:15:19.884 DEBUG:teuthology.orchestra.run.smithi163:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-24T13:15:19.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T13:15:19.925 DEBUG:teuthology.orchestra.run.smithi187:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-24T13:15:19.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T13:15:19.944 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-24T13:15:19.946 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-24T13:15:19.968 DEBUG:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-24T13:15:19.982 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-24T13:15:20.000 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-24T13:15:20.003 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-24T13:15:20.018 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-24T13:15:20.031 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-24T13:15:20.052 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-24T13:15:20.054 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-24T13:15:20.060 INFO:teuthology.task.internal:Configuring sudo... 2024-01-24T13:15:20.061 DEBUG:teuthology.orchestra.run.smithi029:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-24T13:15:20.063 DEBUG:teuthology.orchestra.run.smithi163:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-24T13:15:20.066 DEBUG:teuthology.orchestra.run.smithi187:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-24T13:15:20.096 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-24T13:15:20.105 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-24T13:15:20.105 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-24T13:15:20.138 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-24T13:15:20.140 DEBUG:teuthology.orchestra.run.smithi187:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-24T13:15:20.158 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-24T13:15:20.247 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-24T13:15:20.309 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-24T13:15:20.399 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-24T13:15:20.448 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-24T13:15:20.449 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-24T13:15:20.522 DEBUG:teuthology.orchestra.run.smithi163:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-24T13:15:20.572 DEBUG:teuthology.orchestra.run.smithi163:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-24T13:15:20.625 DEBUG:teuthology.orchestra.run.smithi163:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-24T13:15:20.710 DEBUG:teuthology.orchestra.run.smithi163:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-24T13:15:20.754 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-01-24T13:15:20.754 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-24T13:15:20.822 DEBUG:teuthology.orchestra.run.smithi187:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-24T13:15:20.871 DEBUG:teuthology.orchestra.run.smithi187:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-24T13:15:20.928 DEBUG:teuthology.orchestra.run.smithi187:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-24T13:15:21.013 DEBUG:teuthology.orchestra.run.smithi187:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-24T13:15:21.060 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-01-24T13:15:21.060 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-24T13:15:21.129 DEBUG:teuthology.orchestra.run.smithi029:> sudo service rsyslog restart 2024-01-24T13:15:21.132 DEBUG:teuthology.orchestra.run.smithi163:> sudo service rsyslog restart 2024-01-24T13:15:21.135 DEBUG:teuthology.orchestra.run.smithi187:> sudo service rsyslog restart 2024-01-24T13:15:21.184 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-24T13:15:21.197 INFO:teuthology.orchestra.run.smithi163.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-24T13:15:21.219 INFO:teuthology.orchestra.run.smithi187.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-24T13:15:21.613 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-24T13:15:21.621 INFO:teuthology.task.internal:Starting timer... 2024-01-24T13:15:21.621 INFO:teuthology.run_tasks:Running task pcp... 2024-01-24T13:15:21.690 INFO:teuthology.run_tasks:Running task selinux... 2024-01-24T13:15:21.698 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']} 2024-01-24T13:15:21.699 DEBUG:teuthology.orchestra.run.smithi029:> sudo service auditd rotate 2024-01-24T13:15:21.795 INFO:teuthology.orchestra.run.smithi029.stdout:Rotating logs: 2024-01-24T13:15:21.797 DEBUG:teuthology.orchestra.run.smithi163:> sudo service auditd rotate 2024-01-24T13:15:21.859 INFO:teuthology.orchestra.run.smithi163.stdout:Rotating logs: 2024-01-24T13:15:21.861 DEBUG:teuthology.orchestra.run.smithi187:> sudo service auditd rotate 2024-01-24T13:15:21.950 INFO:teuthology.orchestra.run.smithi187.stdout:Rotating logs: 2024-01-24T13:15:21.953 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-24T13:15:21.953 DEBUG:teuthology.orchestra.run.smithi029:> /usr/sbin/getenforce 2024-01-24T13:15:21.985 INFO:teuthology.orchestra.run.smithi029.stdout:Permissive 2024-01-24T13:15:21.986 DEBUG:teuthology.orchestra.run.smithi163:> /usr/sbin/getenforce 2024-01-24T13:15:22.011 INFO:teuthology.orchestra.run.smithi163.stdout:Permissive 2024-01-24T13:15:22.011 DEBUG:teuthology.orchestra.run.smithi187:> /usr/sbin/getenforce 2024-01-24T13:15:22.040 INFO:teuthology.orchestra.run.smithi187.stdout:Permissive 2024-01-24T13:15:22.040 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi029.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi163.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi187.front.sepia.ceph.com': 'permissive'} 2024-01-24T13:15:22.040 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-24T13:15:22.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T13:15:22.072 DEBUG:teuthology.orchestra.run.smithi163:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-24T13:15:22.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T13:15:22.101 DEBUG:teuthology.orchestra.run.smithi187:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-24T13:15:22.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T13:15:22.132 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-24T13:15:22.132 DEBUG:teuthology.orchestra.run.smithi029:> sudo /usr/sbin/setenforce permissive 2024-01-24T13:15:22.164 DEBUG:teuthology.orchestra.run.smithi163:> sudo /usr/sbin/setenforce permissive 2024-01-24T13:15:22.196 DEBUG:teuthology.orchestra.run.smithi187:> sudo /usr/sbin/setenforce permissive 2024-01-24T13:15:22.223 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-24T13:15:22.233 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-01-24T13:15:22.239 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-01-24T13:15:22.445 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-24T13:15:22.461 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-24T13:15:22.462 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi029.front.sepia.ceph.com,smithi163.front.sepia.ceph.com,smithi187.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-24T13:24:59.268 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi029.front.sepia.ceph.com'), Remote(name='ubuntu@smithi163.front.sepia.ceph.com'), Remote(name='ubuntu@smithi187.front.sepia.ceph.com')] 2024-01-24T13:24:59.269 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-01-24T13:24:59.270 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-24T13:24:59.350 DEBUG:teuthology.orchestra.run.smithi029:> true 2024-01-24T13:24:59.425 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-01-24T13:24:59.425 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-01-24T13:24:59.426 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-24T13:24:59.499 DEBUG:teuthology.orchestra.run.smithi163:> true 2024-01-24T13:24:59.571 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-01-24T13:24:59.571 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-01-24T13:24:59.572 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-24T13:24:59.647 DEBUG:teuthology.orchestra.run.smithi187:> true 2024-01-24T13:24:59.720 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-01-24T13:24:59.720 INFO:teuthology.run_tasks:Running task clock... 2024-01-24T13:24:59.732 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-24T13:24:59.732 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-24T13:24:59.733 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-24T13:24:59.735 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-24T13:24:59.736 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-24T13:24:59.738 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-24T13:24:59.738 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-24T13:24:59.771 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-24T13:24:59.773 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-24T13:24:59.786 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-24T13:24:59.792 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-24T13:24:59.797 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-24T13:24:59.814 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-24T13:24:59.860 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: ntpd: command not found 2024-01-24T13:24:59.873 INFO:teuthology.orchestra.run.smithi163.stdout:506 Cannot talk to daemon 2024-01-24T13:24:59.876 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: ntpd: command not found 2024-01-24T13:24:59.888 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-24T13:24:59.891 INFO:teuthology.orchestra.run.smithi029.stdout:506 Cannot talk to daemon 2024-01-24T13:24:59.896 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: ntpd: command not found 2024-01-24T13:24:59.902 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-24T13:24:59.908 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-24T13:24:59.909 INFO:teuthology.orchestra.run.smithi187.stdout:506 Cannot talk to daemon 2024-01-24T13:24:59.925 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-24T13:24:59.926 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-24T13:24:59.940 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-24T13:24:59.979 INFO:teuthology.orchestra.run.smithi163.stderr:bash: ntpq: command not found 2024-01-24T13:24:59.983 INFO:teuthology.orchestra.run.smithi163.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-24T13:24:59.983 INFO:teuthology.orchestra.run.smithi163.stdout:=============================================================================== 2024-01-24T13:24:59.983 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-24T13:24:59.983 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-24T13:24:59.983 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-24T13:24:59.983 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-24T13:25:00.011 INFO:teuthology.orchestra.run.smithi029.stderr:bash: ntpq: command not found 2024-01-24T13:25:00.014 INFO:teuthology.orchestra.run.smithi187.stderr:bash: ntpq: command not found 2024-01-24T13:25:00.014 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-24T13:25:00.015 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2024-01-24T13:25:00.015 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-24T13:25:00.015 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-24T13:25:00.015 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-24T13:25:00.015 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-24T13:25:00.017 INFO:teuthology.orchestra.run.smithi187.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-24T13:25:00.017 INFO:teuthology.orchestra.run.smithi187.stdout:=============================================================================== 2024-01-24T13:25:00.018 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-24T13:25:00.018 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-24T13:25:00.018 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-24T13:25:00.018 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-24T13:25:00.019 INFO:teuthology.run_tasks:Running task pexec... 2024-01-24T13:25:00.029 INFO:teuthology.task.pexec:Executing custom commands... 2024-01-24T13:25:00.029 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi029.front.sepia.ceph.com 2024-01-24T13:25:00.029 DEBUG:teuthology.orchestra.run.smithi029:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-24T13:25:00.030 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi163.front.sepia.ceph.com 2024-01-24T13:25:00.030 DEBUG:teuthology.orchestra.run.smithi163:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-24T13:25:00.030 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi187.front.sepia.ceph.com 2024-01-24T13:25:00.031 DEBUG:teuthology.orchestra.run.smithi187:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-24T13:25:00.407 INFO:teuthology.orchestra.run.smithi163.stdout:Updating Subscription Management repositories. 2024-01-24T13:25:00.439 INFO:teuthology.orchestra.run.smithi187.stdout:Updating Subscription Management repositories. 2024-01-24T13:25:00.493 INFO:teuthology.orchestra.run.smithi029.stdout:Updating Subscription Management repositories. 2024-01-24T13:25:02.379 INFO:teuthology.orchestra.run.smithi187.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 18 kB/s | 4.5 kB 00:00 2024-01-24T13:25:02.431 INFO:teuthology.orchestra.run.smithi163.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 22 kB/s | 4.5 kB 00:00 2024-01-24T13:25:02.472 INFO:teuthology.orchestra.run.smithi029.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 17 kB/s | 4.5 kB 00:00 2024-01-24T13:25:02.977 INFO:teuthology.orchestra.run.smithi187.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 37 kB/s | 4.1 kB 00:00 2024-01-24T13:25:03.026 INFO:teuthology.orchestra.run.smithi163.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 65 kB/s | 4.1 kB 00:00 2024-01-24T13:25:03.179 INFO:teuthology.orchestra.run.smithi029.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 39 kB/s | 4.1 kB 00:00 2024-01-24T13:25:05.807 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-01-24T13:25:05.808 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-01-24T13:25:05.808 INFO:teuthology.orchestra.run.smithi163.stdout: Package Architecture Version Repository Size 2024-01-24T13:25:05.808 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-01-24T13:25:05.808 INFO:teuthology.orchestra.run.smithi163.stdout:Resetting modules: 2024-01-24T13:25:05.808 INFO:teuthology.orchestra.run.smithi163.stdout: container-tools 2024-01-24T13:25:05.808 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-01-24T13:25:05.808 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-01-24T13:25:05.808 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-01-24T13:25:05.808 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-01-24T13:25:05.983 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-01-24T13:25:05.984 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-01-24T13:25:05.984 INFO:teuthology.orchestra.run.smithi187.stdout: Package Architecture Version Repository Size 2024-01-24T13:25:05.984 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-01-24T13:25:05.984 INFO:teuthology.orchestra.run.smithi187.stdout:Resetting modules: 2024-01-24T13:25:05.985 INFO:teuthology.orchestra.run.smithi187.stdout: container-tools 2024-01-24T13:25:05.985 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-01-24T13:25:05.985 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-01-24T13:25:05.985 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-01-24T13:25:05.985 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-01-24T13:25:06.435 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-24T13:25:06.436 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-24T13:25:06.436 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-01-24T13:25:06.436 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-24T13:25:06.437 INFO:teuthology.orchestra.run.smithi029.stdout:Resetting modules: 2024-01-24T13:25:06.437 INFO:teuthology.orchestra.run.smithi029.stdout: container-tools 2024-01-24T13:25:06.437 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:25:06.437 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-01-24T13:25:06.437 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-24T13:25:06.437 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:25:18.991 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-01-24T13:25:19.298 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-24T13:25:19.393 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-01-24T13:25:19.458 INFO:teuthology.orchestra.run.smithi163.stdout:Updating Subscription Management repositories. 2024-01-24T13:25:19.828 INFO:teuthology.orchestra.run.smithi029.stdout:Updating Subscription Management repositories. 2024-01-24T13:25:19.877 INFO:teuthology.orchestra.run.smithi187.stdout:Updating Subscription Management repositories. 2024-01-24T13:25:21.365 INFO:teuthology.orchestra.run.smithi163.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 34 kB/s | 4.5 kB 00:00 2024-01-24T13:25:21.658 INFO:teuthology.orchestra.run.smithi187.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 34 kB/s | 4.5 kB 00:00 2024-01-24T13:25:21.714 INFO:teuthology.orchestra.run.smithi029.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 28 kB/s | 4.5 kB 00:00 2024-01-24T13:25:22.044 INFO:teuthology.orchestra.run.smithi163.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 28 kB/s | 4.1 kB 00:00 2024-01-24T13:25:22.320 INFO:teuthology.orchestra.run.smithi187.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 25 kB/s | 4.1 kB 00:00 2024-01-24T13:25:22.487 INFO:teuthology.orchestra.run.smithi029.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 22 kB/s | 4.1 kB 00:00 2024-01-24T13:25:24.808 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-01-24T13:25:24.967 INFO:teuthology.orchestra.run.smithi163.stdout:========================================================================================================== 2024-01-24T13:25:24.968 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-01-24T13:25:24.968 INFO:teuthology.orchestra.run.smithi163.stdout:========================================================================================================== 2024-01-24T13:25:24.968 INFO:teuthology.orchestra.run.smithi163.stdout:Installing group/module packages: 2024-01-24T13:25:24.968 INFO:teuthology.orchestra.run.smithi163.stdout: aardvark-dns x86_64 2:1.7.0-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 1.0 M 2024-01-24T13:25:24.968 INFO:teuthology.orchestra.run.smithi163.stdout: buildah x86_64 1:1.31.3-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 8.8 M 2024-01-24T13:25:24.968 INFO:teuthology.orchestra.run.smithi163.stdout: cockpit-podman noarch 75-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 739 k 2024-01-24T13:25:24.968 INFO:teuthology.orchestra.run.smithi163.stdout: crun x86_64 1.8.7-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 239 k 2024-01-24T13:25:24.968 INFO:teuthology.orchestra.run.smithi163.stdout: netavark x86_64 2:1.7.0-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-01-24T13:25:24.968 INFO:teuthology.orchestra.run.smithi163.stdout: python3-podman noarch 4.6.0-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 153 k 2024-01-24T13:25:24.968 INFO:teuthology.orchestra.run.smithi163.stdout: skopeo x86_64 2:1.13.3-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 8.1 M 2024-01-24T13:25:24.968 INFO:teuthology.orchestra.run.smithi163.stdout: toolbox x86_64 0.0.99.4-5.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 2.5 M 2024-01-24T13:25:24.968 INFO:teuthology.orchestra.run.smithi163.stdout: udica noarch 0.2.6-20.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 49 k 2024-01-24T13:25:24.968 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-01-24T13:25:24.969 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2024-01-24T13:25:24.969 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2024-01-24T13:25:24.969 INFO:teuthology.orchestra.run.smithi163.stdout: yajl x86_64 2.1.0-12.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2024-01-24T13:25:24.969 INFO:teuthology.orchestra.run.smithi163.stdout:Installing weak dependencies: 2024-01-24T13:25:24.969 INFO:teuthology.orchestra.run.smithi163.stdout: criu-libs x86_64 3.18-4.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 39 k 2024-01-24T13:25:24.969 INFO:teuthology.orchestra.run.smithi163.stdout:Installing module profiles: 2024-01-24T13:25:24.969 INFO:teuthology.orchestra.run.smithi163.stdout: container-tools/common 2024-01-24T13:25:24.969 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-01-24T13:25:24.969 INFO:teuthology.orchestra.run.smithi163.stdout:Enabling module streams: 2024-01-24T13:25:24.969 INFO:teuthology.orchestra.run.smithi163.stdout: container-tools rhel8 2024-01-24T13:25:24.969 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-01-24T13:25:24.969 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-01-24T13:25:24.969 INFO:teuthology.orchestra.run.smithi163.stdout:========================================================================================================== 2024-01-24T13:25:24.969 INFO:teuthology.orchestra.run.smithi163.stdout:Install 13 Packages 2024-01-24T13:25:24.969 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-01-24T13:25:24.970 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 26 M 2024-01-24T13:25:24.970 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 87 M 2024-01-24T13:25:24.970 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-01-24T13:25:25.063 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-01-24T13:25:25.176 INFO:teuthology.orchestra.run.smithi163.stdout:(1/13): python3-pytoml-0.1.14-5.git7dea353.el8. 121 kB/s | 25 kB 00:00 2024-01-24T13:25:25.214 INFO:teuthology.orchestra.run.smithi163.stdout:(2/13): yajl-2.1.0-12.el8.x86_64.rpm 168 kB/s | 41 kB 00:00 2024-01-24T13:25:25.216 INFO:teuthology.orchestra.run.smithi187.stdout:========================================================================================================== 2024-01-24T13:25:25.216 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-01-24T13:25:25.216 INFO:teuthology.orchestra.run.smithi187.stdout:========================================================================================================== 2024-01-24T13:25:25.216 INFO:teuthology.orchestra.run.smithi187.stdout:Installing group/module packages: 2024-01-24T13:25:25.217 INFO:teuthology.orchestra.run.smithi187.stdout: aardvark-dns x86_64 2:1.7.0-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 1.0 M 2024-01-24T13:25:25.217 INFO:teuthology.orchestra.run.smithi187.stdout: buildah x86_64 1:1.31.3-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 8.8 M 2024-01-24T13:25:25.217 INFO:teuthology.orchestra.run.smithi187.stdout: cockpit-podman noarch 75-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 739 k 2024-01-24T13:25:25.217 INFO:teuthology.orchestra.run.smithi187.stdout: crun x86_64 1.8.7-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 239 k 2024-01-24T13:25:25.217 INFO:teuthology.orchestra.run.smithi187.stdout: netavark x86_64 2:1.7.0-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-01-24T13:25:25.217 INFO:teuthology.orchestra.run.smithi187.stdout: python3-podman noarch 4.6.0-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 153 k 2024-01-24T13:25:25.217 INFO:teuthology.orchestra.run.smithi187.stdout: skopeo x86_64 2:1.13.3-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 8.1 M 2024-01-24T13:25:25.217 INFO:teuthology.orchestra.run.smithi187.stdout: toolbox x86_64 0.0.99.4-5.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 2.5 M 2024-01-24T13:25:25.217 INFO:teuthology.orchestra.run.smithi187.stdout: udica noarch 0.2.6-20.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 49 k 2024-01-24T13:25:25.217 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2024-01-24T13:25:25.217 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2024-01-24T13:25:25.217 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2024-01-24T13:25:25.217 INFO:teuthology.orchestra.run.smithi187.stdout: yajl x86_64 2.1.0-12.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2024-01-24T13:25:25.217 INFO:teuthology.orchestra.run.smithi187.stdout:Installing weak dependencies: 2024-01-24T13:25:25.217 INFO:teuthology.orchestra.run.smithi187.stdout: criu-libs x86_64 3.18-4.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 39 k 2024-01-24T13:25:25.218 INFO:teuthology.orchestra.run.smithi187.stdout:Installing module profiles: 2024-01-24T13:25:25.218 INFO:teuthology.orchestra.run.smithi187.stdout: container-tools/common 2024-01-24T13:25:25.218 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-01-24T13:25:25.218 INFO:teuthology.orchestra.run.smithi187.stdout:Enabling module streams: 2024-01-24T13:25:25.218 INFO:teuthology.orchestra.run.smithi187.stdout: container-tools rhel8 2024-01-24T13:25:25.218 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-01-24T13:25:25.218 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-01-24T13:25:25.218 INFO:teuthology.orchestra.run.smithi187.stdout:========================================================================================================== 2024-01-24T13:25:25.218 INFO:teuthology.orchestra.run.smithi187.stdout:Install 13 Packages 2024-01-24T13:25:25.218 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-01-24T13:25:25.219 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 26 M 2024-01-24T13:25:25.219 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 87 M 2024-01-24T13:25:25.219 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-01-24T13:25:25.239 INFO:teuthology.orchestra.run.smithi163.stdout:(3/13): python3-pyxdg-0.25-16.el8.noarch.rpm 350 kB/s | 94 kB 00:00 2024-01-24T13:25:25.356 INFO:teuthology.orchestra.run.smithi163.stdout:(4/13): aardvark-dns-1.7.0-1.module+el8.9.0+203 5.6 MB/s | 1.0 MB 00:00 2024-01-24T13:25:25.358 INFO:teuthology.orchestra.run.smithi187.stdout:(1/13): python3-pytoml-0.1.14-5.git7dea353.el8. 179 kB/s | 25 kB 00:00 2024-01-24T13:25:25.400 INFO:teuthology.orchestra.run.smithi187.stdout:(2/13): yajl-2.1.0-12.el8.x86_64.rpm 227 kB/s | 41 kB 00:00 2024-01-24T13:25:25.434 INFO:teuthology.orchestra.run.smithi187.stdout:(3/13): python3-pyxdg-0.25-16.el8.noarch.rpm 438 kB/s | 94 kB 00:00 2024-01-24T13:25:25.515 INFO:teuthology.orchestra.run.smithi163.stdout:(5/13): criu-libs-3.18-4.module+el8.9.0+20326+3 248 kB/s | 39 kB 00:00 2024-01-24T13:25:25.548 INFO:teuthology.orchestra.run.smithi163.stdout:(6/13): cockpit-podman-75-1.module+el8.9.0+2032 2.3 MB/s | 739 kB 00:00 2024-01-24T13:25:25.651 INFO:teuthology.orchestra.run.smithi187.stdout:(4/13): buildah-1.31.3-1.module+el8.9.0+20326+3 35 MB/s | 8.8 MB 00:00 2024-01-24T13:25:25.684 INFO:teuthology.orchestra.run.smithi187.stdout:(5/13): cockpit-podman-75-1.module+el8.9.0+2032 2.9 MB/s | 739 kB 00:00 2024-01-24T13:25:25.698 INFO:teuthology.orchestra.run.smithi163.stdout:(7/13): buildah-1.31.3-1.module+el8.9.0+20326+3 18 MB/s | 8.8 MB 00:00 2024-01-24T13:25:25.718 INFO:teuthology.orchestra.run.smithi187.stdout:(6/13): aardvark-dns-1.7.0-1.module+el8.9.0+203 2.8 MB/s | 1.0 MB 00:00 2024-01-24T13:25:25.724 INFO:teuthology.orchestra.run.smithi163.stdout:(8/13): crun-1.8.7-1.module+el8.9.0+20326+38708 1.1 MB/s | 239 kB 00:00 2024-01-24T13:25:25.754 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-24T13:25:25.785 INFO:teuthology.orchestra.run.smithi187.stdout:(7/13): criu-libs-3.18-4.module+el8.9.0+20326+3 295 kB/s | 39 kB 00:00 2024-01-24T13:25:25.818 INFO:teuthology.orchestra.run.smithi187.stdout:(8/13): crun-1.8.7-1.module+el8.9.0+20326+38708 1.7 MB/s | 239 kB 00:00 2024-01-24T13:25:25.841 INFO:teuthology.orchestra.run.smithi163.stdout:(9/13): netavark-1.7.0-1.module+el8.9.0+20326+3 13 MB/s | 3.7 MB 00:00 2024-01-24T13:25:25.867 INFO:teuthology.orchestra.run.smithi163.stdout:(10/13): python3-podman-4.6.0-1.module+el8.9.0+ 917 kB/s | 153 kB 00:00 2024-01-24T13:25:25.934 INFO:teuthology.orchestra.run.smithi029.stdout:========================================================================================================== 2024-01-24T13:25:25.934 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-01-24T13:25:25.934 INFO:teuthology.orchestra.run.smithi029.stdout:========================================================================================================== 2024-01-24T13:25:25.934 INFO:teuthology.orchestra.run.smithi029.stdout:Installing group/module packages: 2024-01-24T13:25:25.935 INFO:teuthology.orchestra.run.smithi029.stdout: aardvark-dns x86_64 2:1.7.0-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 1.0 M 2024-01-24T13:25:25.935 INFO:teuthology.orchestra.run.smithi029.stdout: buildah x86_64 1:1.31.3-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 8.8 M 2024-01-24T13:25:25.935 INFO:teuthology.orchestra.run.smithi029.stdout: cockpit-podman noarch 75-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 739 k 2024-01-24T13:25:25.935 INFO:teuthology.orchestra.run.smithi029.stdout: crun x86_64 1.8.7-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 239 k 2024-01-24T13:25:25.935 INFO:teuthology.orchestra.run.smithi029.stdout: netavark x86_64 2:1.7.0-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-01-24T13:25:25.935 INFO:teuthology.orchestra.run.smithi029.stdout: python3-podman noarch 4.6.0-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 153 k 2024-01-24T13:25:25.935 INFO:teuthology.orchestra.run.smithi029.stdout: skopeo x86_64 2:1.13.3-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 8.1 M 2024-01-24T13:25:25.935 INFO:teuthology.orchestra.run.smithi029.stdout: toolbox x86_64 0.0.99.4-5.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 2.5 M 2024-01-24T13:25:25.935 INFO:teuthology.orchestra.run.smithi029.stdout: udica noarch 0.2.6-20.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 49 k 2024-01-24T13:25:25.935 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-01-24T13:25:25.935 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2024-01-24T13:25:25.935 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2024-01-24T13:25:25.935 INFO:teuthology.orchestra.run.smithi029.stdout: yajl x86_64 2.1.0-12.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2024-01-24T13:25:25.935 INFO:teuthology.orchestra.run.smithi029.stdout:Installing weak dependencies: 2024-01-24T13:25:25.935 INFO:teuthology.orchestra.run.smithi029.stdout: criu-libs x86_64 3.18-4.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 39 k 2024-01-24T13:25:25.935 INFO:teuthology.orchestra.run.smithi029.stdout:Installing module profiles: 2024-01-24T13:25:25.936 INFO:teuthology.orchestra.run.smithi029.stdout: container-tools/common 2024-01-24T13:25:25.936 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:25:25.936 INFO:teuthology.orchestra.run.smithi029.stdout:Enabling module streams: 2024-01-24T13:25:25.936 INFO:teuthology.orchestra.run.smithi029.stdout: container-tools rhel8 2024-01-24T13:25:25.936 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:25:25.936 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-01-24T13:25:25.936 INFO:teuthology.orchestra.run.smithi029.stdout:========================================================================================================== 2024-01-24T13:25:25.936 INFO:teuthology.orchestra.run.smithi029.stdout:Install 13 Packages 2024-01-24T13:25:25.936 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:25:25.937 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 26 M 2024-01-24T13:25:25.937 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 87 M 2024-01-24T13:25:25.937 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-01-24T13:25:25.952 INFO:teuthology.orchestra.run.smithi187.stdout:(9/13): netavark-1.7.0-1.module+el8.9.0+20326+3 16 MB/s | 3.7 MB 00:00 2024-01-24T13:25:25.977 INFO:teuthology.orchestra.run.smithi187.stdout:(10/13): python3-podman-4.6.0-1.module+el8.9.0+ 798 kB/s | 153 kB 00:00 2024-01-24T13:25:26.042 INFO:teuthology.orchestra.run.smithi163.stdout:(11/13): skopeo-1.13.3-1.module+el8.9.0+20326+3 26 MB/s | 8.1 MB 00:00 2024-01-24T13:25:26.058 INFO:teuthology.orchestra.run.smithi163.stdout:(12/13): udica-0.2.6-20.module+el8.9.0+20326+38 258 kB/s | 49 kB 00:00 2024-01-24T13:25:26.111 INFO:teuthology.orchestra.run.smithi187.stdout:(11/13): toolbox-0.0.99.4-5.module+el8.9.0+2032 16 MB/s | 2.5 MB 00:00 2024-01-24T13:25:26.121 INFO:teuthology.orchestra.run.smithi029.stdout:(1/13): yajl-2.1.0-12.el8.x86_64.rpm 224 kB/s | 41 kB 00:00 2024-01-24T13:25:26.128 INFO:teuthology.orchestra.run.smithi187.stdout:(12/13): udica-0.2.6-20.module+el8.9.0+20326+38 329 kB/s | 49 kB 00:00 2024-01-24T13:25:26.134 INFO:teuthology.orchestra.run.smithi163.stdout:(13/13): toolbox-0.0.99.4-5.module+el8.9.0+2032 8.7 MB/s | 2.5 MB 00:00 2024-01-24T13:25:26.134 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-01-24T13:25:26.134 INFO:teuthology.orchestra.run.smithi163.stdout:Total 22 MB/s | 26 MB 00:01 2024-01-24T13:25:26.146 INFO:teuthology.orchestra.run.smithi029.stdout:(2/13): python3-pytoml-0.1.14-5.git7dea353.el8. 119 kB/s | 25 kB 00:00 2024-01-24T13:25:26.172 INFO:teuthology.orchestra.run.smithi029.stdout:(3/13): python3-pyxdg-0.25-16.el8.noarch.rpm 401 kB/s | 94 kB 00:00 2024-01-24T13:25:26.244 INFO:teuthology.orchestra.run.smithi187.stdout:(13/13): skopeo-1.13.3-1.module+el8.9.0+20326+3 19 MB/s | 8.1 MB 00:00 2024-01-24T13:25:26.245 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-01-24T13:25:26.245 INFO:teuthology.orchestra.run.smithi187.stdout:Total 25 MB/s | 26 MB 00:01 2024-01-24T13:25:26.273 INFO:teuthology.orchestra.run.smithi029.stdout:(4/13): aardvark-dns-1.7.0-1.module+el8.9.0+203 6.7 MB/s | 1.0 MB 00:00 2024-01-24T13:25:26.324 INFO:teuthology.orchestra.run.smithi029.stdout:(5/13): cockpit-podman-75-1.module+el8.9.0+2032 4.8 MB/s | 739 kB 00:00 2024-01-24T13:25:26.382 INFO:teuthology.orchestra.run.smithi029.stdout:(6/13): criu-libs-3.18-4.module+el8.9.0+20326+3 361 kB/s | 39 kB 00:00 2024-01-24T13:25:26.419 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-01-24T13:25:26.440 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-01-24T13:25:26.440 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-01-24T13:25:26.465 INFO:teuthology.orchestra.run.smithi029.stdout:(7/13): crun-1.8.7-1.module+el8.9.0+20326+38708 1.6 MB/s | 239 kB 00:00 2024-01-24T13:25:26.531 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-01-24T13:25:26.553 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-01-24T13:25:26.553 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-01-24T13:25:26.567 INFO:teuthology.orchestra.run.smithi029.stdout:(8/13): python3-podman-4.6.0-1.module+el8.9.0+2 1.5 MB/s | 153 kB 00:00 2024-01-24T13:25:26.668 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-01-24T13:25:26.677 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-01-24T13:25:26.751 INFO:teuthology.orchestra.run.smithi029.stdout:(9/13): buildah-1.31.3-1.module+el8.9.0+20326+3 15 MB/s | 8.8 MB 00:00 2024-01-24T13:25:26.783 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-01-24T13:25:26.791 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-01-24T13:25:26.885 INFO:teuthology.orchestra.run.smithi029.stdout:(10/13): netavark-1.7.0-1.module+el8.9.0+20326+ 7.4 MB/s | 3.7 MB 00:00 2024-01-24T13:25:27.042 INFO:teuthology.orchestra.run.smithi029.stdout:(11/13): toolbox-0.0.99.4-5.module+el8.9.0+2032 8.7 MB/s | 2.5 MB 00:00 2024-01-24T13:25:27.069 INFO:teuthology.orchestra.run.smithi029.stdout:(12/13): udica-0.2.6-20.module+el8.9.0+20326+38 269 kB/s | 49 kB 00:00 2024-01-24T13:25:27.228 INFO:teuthology.orchestra.run.smithi029.stdout:(13/13): skopeo-1.13.3-1.module+el8.9.0+20326+3 12 MB/s | 8.1 MB 00:00 2024-01-24T13:25:27.229 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-01-24T13:25:27.229 INFO:teuthology.orchestra.run.smithi029.stdout:Total 20 MB/s | 26 MB 00:01 2024-01-24T13:25:27.315 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-01-24T13:25:27.403 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-01-24T13:25:27.545 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-01-24T13:25:27.548 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 1/13 2024-01-24T13:25:27.569 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-01-24T13:25:27.570 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-01-24T13:25:27.627 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 1/13 2024-01-24T13:25:27.827 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-01-24T13:25:27.837 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-01-24T13:25:28.083 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : aardvark-dns-2:1.7.0-1.module+el8.9.0+20326+387084 2/13 2024-01-24T13:25:28.121 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : aardvark-dns-2:1.7.0-1.module+el8.9.0+20326+387084 2/13 2024-01-24T13:25:28.292 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : netavark-2:1.7.0-1.module+el8.9.0+20326+387084d0.x 3/13 2024-01-24T13:25:28.337 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : netavark-2:1.7.0-1.module+el8.9.0+20326+387084d0.x 3/13 2024-01-24T13:25:28.443 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-01-24T13:25:28.466 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : yajl-2.1.0-12.el8.x86_64 4/13 2024-01-24T13:25:28.529 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : yajl-2.1.0-12.el8.x86_64 4/13 2024-01-24T13:25:28.662 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 5/13 2024-01-24T13:25:28.721 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 1/13 2024-01-24T13:25:28.749 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 5/13 2024-01-24T13:25:28.862 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 6/13 2024-01-24T13:25:28.975 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 6/13 2024-01-24T13:25:29.098 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-podman-4.6.0-1.module+el8.9.0+20326+387084 7/13 2024-01-24T13:25:29.236 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-podman-4.6.0-1.module+el8.9.0+20326+387084 7/13 2024-01-24T13:25:29.258 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : aardvark-dns-2:1.7.0-1.module+el8.9.0+20326+387084 2/13 2024-01-24T13:25:29.459 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : netavark-2:1.7.0-1.module+el8.9.0+20326+387084d0.x 3/13 2024-01-24T13:25:29.652 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : yajl-2.1.0-12.el8.x86_64 4/13 2024-01-24T13:25:29.828 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 5/13 2024-01-24T13:25:30.028 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : crun-1.8.7-1.module+el8.9.0+20326+387084d0.x86_64 8/13 2024-01-24T13:25:30.095 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 6/13 2024-01-24T13:25:30.116 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : crun-1.8.7-1.module+el8.9.0+20326+387084d0.x86_64 8/13 2024-01-24T13:25:30.256 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : buildah-1:1.31.3-1.module+el8.9.0+20326+387084d0.x 9/13 2024-01-24T13:25:30.331 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-podman-4.6.0-1.module+el8.9.0+20326+387084 7/13 2024-01-24T13:25:30.334 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : buildah-1:1.31.3-1.module+el8.9.0+20326+387084d0.x 9/13 2024-01-24T13:25:30.658 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : udica-0.2.6-20.module+el8.9.0+20326+387084d0.noarc 10/13 2024-01-24T13:25:30.703 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : udica-0.2.6-20.module+el8.9.0+20326+387084d0.noarc 10/13 2024-01-24T13:25:31.297 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : crun-1.8.7-1.module+el8.9.0+20326+387084d0.x86_64 8/13 2024-01-24T13:25:31.565 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : buildah-1:1.31.3-1.module+el8.9.0+20326+387084d0.x 9/13 2024-01-24T13:25:31.590 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : toolbox-0.0.99.4-5.module+el8.9.0+20326+387084d0.x 11/13 2024-01-24T13:25:31.592 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : toolbox-0.0.99.4-5.module+el8.9.0+20326+387084d0.x 11/13 2024-01-24T13:25:31.839 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : skopeo-2:1.13.3-1.module+el8.9.0+20326+387084d0.x8 12/13 2024-01-24T13:25:31.852 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : skopeo-2:1.13.3-1.module+el8.9.0+20326+387084d0.x8 12/13 2024-01-24T13:25:31.962 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : udica-0.2.6-20.module+el8.9.0+20326+387084d0.noarc 10/13 2024-01-24T13:25:32.047 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : cockpit-podman-75-1.module+el8.9.0+20326+387084d0. 13/13 2024-01-24T13:25:32.052 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : cockpit-podman-75-1.module+el8.9.0+20326+387084d0. 13/13 2024-01-24T13:25:32.470 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: cockpit-podman-75-1.module+el8.9.0+20326+387084d0. 13/13 2024-01-24T13:25:32.470 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/13 2024-01-24T13:25:32.470 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-01-24T13:25:32.470 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 3/13 2024-01-24T13:25:32.470 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : aardvark-dns-2:1.7.0-1.module+el8.9.0+20326+387084 4/13 2024-01-24T13:25:32.470 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : buildah-1:1.31.3-1.module+el8.9.0+20326+387084d0.x 5/13 2024-01-24T13:25:32.470 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : cockpit-podman-75-1.module+el8.9.0+20326+387084d0. 6/13 2024-01-24T13:25:32.470 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 7/13 2024-01-24T13:25:32.470 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : crun-1.8.7-1.module+el8.9.0+20326+387084d0.x86_64 8/13 2024-01-24T13:25:32.470 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : netavark-2:1.7.0-1.module+el8.9.0+20326+387084d0.x 9/13 2024-01-24T13:25:32.470 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-podman-4.6.0-1.module+el8.9.0+20326+387084 10/13 2024-01-24T13:25:32.470 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : skopeo-2:1.13.3-1.module+el8.9.0+20326+387084d0.x8 11/13 2024-01-24T13:25:32.471 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : toolbox-0.0.99.4-5.module+el8.9.0+20326+387084d0.x 12/13 2024-01-24T13:25:32.476 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: cockpit-podman-75-1.module+el8.9.0+20326+387084d0. 13/13 2024-01-24T13:25:32.477 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/13 2024-01-24T13:25:32.477 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-01-24T13:25:32.477 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 3/13 2024-01-24T13:25:32.477 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : aardvark-dns-2:1.7.0-1.module+el8.9.0+20326+387084 4/13 2024-01-24T13:25:32.477 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : buildah-1:1.31.3-1.module+el8.9.0+20326+387084d0.x 5/13 2024-01-24T13:25:32.477 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : cockpit-podman-75-1.module+el8.9.0+20326+387084d0. 6/13 2024-01-24T13:25:32.477 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 7/13 2024-01-24T13:25:32.477 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : crun-1.8.7-1.module+el8.9.0+20326+387084d0.x86_64 8/13 2024-01-24T13:25:32.477 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : netavark-2:1.7.0-1.module+el8.9.0+20326+387084d0.x 9/13 2024-01-24T13:25:32.477 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-podman-4.6.0-1.module+el8.9.0+20326+387084 10/13 2024-01-24T13:25:32.477 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : skopeo-2:1.13.3-1.module+el8.9.0+20326+387084d0.x8 11/13 2024-01-24T13:25:32.477 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : toolbox-0.0.99.4-5.module+el8.9.0+20326+387084d0.x 12/13 2024-01-24T13:25:32.930 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : toolbox-0.0.99.4-5.module+el8.9.0+20326+387084d0.x 11/13 2024-01-24T13:25:33.188 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : skopeo-2:1.13.3-1.module+el8.9.0+20326+387084d0.x8 12/13 2024-01-24T13:25:33.375 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cockpit-podman-75-1.module+el8.9.0+20326+387084d0. 13/13 2024-01-24T13:25:33.857 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cockpit-podman-75-1.module+el8.9.0+20326+387084d0. 13/13 2024-01-24T13:25:33.858 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/13 2024-01-24T13:25:33.858 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-01-24T13:25:33.858 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 3/13 2024-01-24T13:25:33.858 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : aardvark-dns-2:1.7.0-1.module+el8.9.0+20326+387084 4/13 2024-01-24T13:25:33.858 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : buildah-1:1.31.3-1.module+el8.9.0+20326+387084d0.x 5/13 2024-01-24T13:25:33.858 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cockpit-podman-75-1.module+el8.9.0+20326+387084d0. 6/13 2024-01-24T13:25:33.858 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 7/13 2024-01-24T13:25:33.858 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : crun-1.8.7-1.module+el8.9.0+20326+387084d0.x86_64 8/13 2024-01-24T13:25:33.858 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : netavark-2:1.7.0-1.module+el8.9.0+20326+387084d0.x 9/13 2024-01-24T13:25:33.858 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-podman-4.6.0-1.module+el8.9.0+20326+387084 10/13 2024-01-24T13:25:33.858 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : skopeo-2:1.13.3-1.module+el8.9.0+20326+387084d0.x8 11/13 2024-01-24T13:25:33.858 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : toolbox-0.0.99.4-5.module+el8.9.0+20326+387084d0.x 12/13 2024-01-24T13:25:44.412 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : udica-0.2.6-20.module+el8.9.0+20326+387084d0.noarc 13/13 2024-01-24T13:25:44.412 INFO:teuthology.orchestra.run.smithi187.stdout:Installed products updated. 2024-01-24T13:25:44.968 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-01-24T13:25:44.968 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-01-24T13:25:44.968 INFO:teuthology.orchestra.run.smithi187.stdout: aardvark-dns-2:1.7.0-1.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T13:25:44.968 INFO:teuthology.orchestra.run.smithi187.stdout: buildah-1:1.31.3-1.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T13:25:44.968 INFO:teuthology.orchestra.run.smithi187.stdout: cockpit-podman-75-1.module+el8.9.0+20326+387084d0.noarch 2024-01-24T13:25:44.968 INFO:teuthology.orchestra.run.smithi187.stdout: criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T13:25:44.968 INFO:teuthology.orchestra.run.smithi187.stdout: crun-1.8.7-1.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T13:25:44.968 INFO:teuthology.orchestra.run.smithi187.stdout: netavark-2:1.7.0-1.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T13:25:44.969 INFO:teuthology.orchestra.run.smithi187.stdout: python3-podman-4.6.0-1.module+el8.9.0+20326+387084d0.noarch 2024-01-24T13:25:44.969 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-01-24T13:25:44.969 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-01-24T13:25:44.969 INFO:teuthology.orchestra.run.smithi187.stdout: skopeo-2:1.13.3-1.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T13:25:44.969 INFO:teuthology.orchestra.run.smithi187.stdout: toolbox-0.0.99.4-5.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T13:25:44.969 INFO:teuthology.orchestra.run.smithi187.stdout: udica-0.2.6-20.module+el8.9.0+20326+387084d0.noarch 2024-01-24T13:25:44.969 INFO:teuthology.orchestra.run.smithi187.stdout: yajl-2.1.0-12.el8.x86_64 2024-01-24T13:25:44.969 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-01-24T13:25:44.969 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-01-24T13:25:45.163 DEBUG:teuthology.parallel:result is None 2024-01-24T13:25:46.988 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : udica-0.2.6-20.module+el8.9.0+20326+387084d0.noarc 13/13 2024-01-24T13:25:46.988 INFO:teuthology.orchestra.run.smithi163.stdout:Installed products updated. 2024-01-24T13:25:47.531 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-01-24T13:25:47.531 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-01-24T13:25:47.531 INFO:teuthology.orchestra.run.smithi163.stdout: aardvark-dns-2:1.7.0-1.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T13:25:47.531 INFO:teuthology.orchestra.run.smithi163.stdout: buildah-1:1.31.3-1.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T13:25:47.531 INFO:teuthology.orchestra.run.smithi163.stdout: cockpit-podman-75-1.module+el8.9.0+20326+387084d0.noarch 2024-01-24T13:25:47.531 INFO:teuthology.orchestra.run.smithi163.stdout: criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T13:25:47.532 INFO:teuthology.orchestra.run.smithi163.stdout: crun-1.8.7-1.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T13:25:47.532 INFO:teuthology.orchestra.run.smithi163.stdout: netavark-2:1.7.0-1.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T13:25:47.532 INFO:teuthology.orchestra.run.smithi163.stdout: python3-podman-4.6.0-1.module+el8.9.0+20326+387084d0.noarch 2024-01-24T13:25:47.532 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-01-24T13:25:47.532 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-01-24T13:25:47.532 INFO:teuthology.orchestra.run.smithi163.stdout: skopeo-2:1.13.3-1.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T13:25:47.532 INFO:teuthology.orchestra.run.smithi163.stdout: toolbox-0.0.99.4-5.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T13:25:47.532 INFO:teuthology.orchestra.run.smithi163.stdout: udica-0.2.6-20.module+el8.9.0+20326+387084d0.noarch 2024-01-24T13:25:47.532 INFO:teuthology.orchestra.run.smithi163.stdout: yajl-2.1.0-12.el8.x86_64 2024-01-24T13:25:47.532 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-01-24T13:25:47.532 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-01-24T13:25:47.723 DEBUG:teuthology.parallel:result is None 2024-01-24T13:25:48.516 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : udica-0.2.6-20.module+el8.9.0+20326+387084d0.noarc 13/13 2024-01-24T13:25:48.516 INFO:teuthology.orchestra.run.smithi029.stdout:Installed products updated. 2024-01-24T13:25:49.124 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:25:49.124 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-01-24T13:25:49.124 INFO:teuthology.orchestra.run.smithi029.stdout: aardvark-dns-2:1.7.0-1.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T13:25:49.124 INFO:teuthology.orchestra.run.smithi029.stdout: buildah-1:1.31.3-1.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T13:25:49.124 INFO:teuthology.orchestra.run.smithi029.stdout: cockpit-podman-75-1.module+el8.9.0+20326+387084d0.noarch 2024-01-24T13:25:49.124 INFO:teuthology.orchestra.run.smithi029.stdout: criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T13:25:49.124 INFO:teuthology.orchestra.run.smithi029.stdout: crun-1.8.7-1.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T13:25:49.125 INFO:teuthology.orchestra.run.smithi029.stdout: netavark-2:1.7.0-1.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T13:25:49.125 INFO:teuthology.orchestra.run.smithi029.stdout: python3-podman-4.6.0-1.module+el8.9.0+20326+387084d0.noarch 2024-01-24T13:25:49.125 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-01-24T13:25:49.125 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-01-24T13:25:49.125 INFO:teuthology.orchestra.run.smithi029.stdout: skopeo-2:1.13.3-1.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T13:25:49.125 INFO:teuthology.orchestra.run.smithi029.stdout: toolbox-0.0.99.4-5.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T13:25:49.125 INFO:teuthology.orchestra.run.smithi029.stdout: udica-0.2.6-20.module+el8.9.0+20326+387084d0.noarch 2024-01-24T13:25:49.125 INFO:teuthology.orchestra.run.smithi029.stdout: yajl-2.1.0-12.el8.x86_64 2024-01-24T13:25:49.125 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:25:49.125 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-24T13:25:49.326 DEBUG:teuthology.parallel:result is None 2024-01-24T13:25:49.326 INFO:teuthology.run_tasks:Running task install... 2024-01-24T13:25:49.335 DEBUG:teuthology.task.install:project ceph 2024-01-24T13:25:49.335 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '4e3182a65800aed6d4aa2f6b86da3cfd72188134'}} 2024-01-24T13:25:49.335 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '4e3182a65800aed6d4aa2f6b86da3cfd72188134'} 2024-01-24T13:25:49.335 INFO:teuthology.task.install:Using flavor: default 2024-01-24T13:25:49.343 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-24T13:25:49.343 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2024-01-24T13:25:49.344 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4e3182a65800aed6d4aa2f6b86da3cfd72188134', 'tag': None, 'wait_for_package': False} 2024-01-24T13:25:49.344 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4e3182a65800aed6d4aa2f6b86da3cfd72188134 2024-01-24T13:25:49.346 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4e3182a65800aed6d4aa2f6b86da3cfd72188134', 'tag': None, 'wait_for_package': False} 2024-01-24T13:25:49.346 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4e3182a65800aed6d4aa2f6b86da3cfd72188134 2024-01-24T13:25:49.347 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4e3182a65800aed6d4aa2f6b86da3cfd72188134', 'tag': None, 'wait_for_package': False} 2024-01-24T13:25:49.347 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4e3182a65800aed6d4aa2f6b86da3cfd72188134 2024-01-24T13:25:49.491 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/8/flavors/default/ 2024-01-24T13:25:49.491 INFO:teuthology.task.install.rpm:Package version is 18.2.1-341.g4e3182a6 2024-01-24T13:25:49.494 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/8/flavors/default/ 2024-01-24T13:25:49.495 INFO:teuthology.task.install.rpm:Package version is 18.2.1-341.g4e3182a6 2024-01-24T13:25:49.611 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-24T13:25:49.612 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-24T13:25:49.612 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-24T13:25:49.647 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-01-24T13:25:49.648 DEBUG:teuthology.orchestra.run.smithi029:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4e3182a65800aed6d4aa2f6b86da3cfd72188134/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-24T13:25:49.668 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/8/flavors/default/ 2024-01-24T13:25:49.668 INFO:teuthology.task.install.rpm:Package version is 18.2.1-341.g4e3182a6 2024-01-24T13:25:49.724 DEBUG:teuthology.orchestra.run.smithi029:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-24T13:25:49.755 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-24T13:25:49.756 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-01-24T13:25:49.756 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-24T13:25:49.788 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-01-24T13:25:49.789 DEBUG:teuthology.orchestra.run.smithi187:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4e3182a65800aed6d4aa2f6b86da3cfd72188134/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-24T13:25:49.812 DEBUG:teuthology.orchestra.run.smithi029:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-24T13:25:49.844 INFO:teuthology.orchestra.run.smithi029.stdout:check_obsoletes = 1 2024-01-24T13:25:49.848 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum clean all 2024-01-24T13:25:49.860 DEBUG:teuthology.orchestra.run.smithi187:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-24T13:25:49.898 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-24T13:25:49.899 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-01-24T13:25:49.899 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-24T13:25:49.932 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-01-24T13:25:49.932 DEBUG:teuthology.orchestra.run.smithi163:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4e3182a65800aed6d4aa2f6b86da3cfd72188134/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-24T13:25:49.942 DEBUG:teuthology.orchestra.run.smithi187:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-24T13:25:50.005 DEBUG:teuthology.orchestra.run.smithi163:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-24T13:25:50.010 INFO:teuthology.orchestra.run.smithi187.stdout:check_obsoletes = 1 2024-01-24T13:25:50.011 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum clean all 2024-01-24T13:25:50.086 DEBUG:teuthology.orchestra.run.smithi163:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-24T13:25:50.155 INFO:teuthology.orchestra.run.smithi163.stdout:check_obsoletes = 1 2024-01-24T13:25:50.156 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum clean all 2024-01-24T13:25:50.282 INFO:teuthology.orchestra.run.smithi029.stdout:Updating Subscription Management repositories. 2024-01-24T13:25:50.387 INFO:teuthology.orchestra.run.smithi187.stdout:Updating Subscription Management repositories. 2024-01-24T13:25:50.535 INFO:teuthology.orchestra.run.smithi163.stdout:Updating Subscription Management repositories. 2024-01-24T13:25:51.532 INFO:teuthology.orchestra.run.smithi187.stdout:57 files removed 2024-01-24T13:25:51.584 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-01-24T13:25:51.719 INFO:teuthology.orchestra.run.smithi029.stdout:57 files removed 2024-01-24T13:25:51.758 INFO:teuthology.orchestra.run.smithi163.stdout:57 files removed 2024-01-24T13:25:51.775 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-01-24T13:25:51.816 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-01-24T13:25:51.920 INFO:teuthology.orchestra.run.smithi187.stdout:Updating Subscription Management repositories. 2024-01-24T13:25:52.153 INFO:teuthology.orchestra.run.smithi163.stdout:Updating Subscription Management repositories. 2024-01-24T13:25:52.162 INFO:teuthology.orchestra.run.smithi029.stdout:Updating Subscription Management repositories. 2024-01-24T13:25:54.073 INFO:teuthology.orchestra.run.smithi187.stdout:ceph packages for x86_64 142 kB/s | 80 kB 00:00 2024-01-24T13:25:54.255 INFO:teuthology.orchestra.run.smithi029.stdout:ceph packages for x86_64 214 kB/s | 80 kB 00:00 2024-01-24T13:25:54.332 INFO:teuthology.orchestra.run.smithi163.stdout:ceph packages for x86_64 152 kB/s | 80 kB 00:00 2024-01-24T13:25:54.432 INFO:teuthology.orchestra.run.smithi187.stdout:ceph noarch packages 29 kB/s | 9.5 kB 00:00 2024-01-24T13:25:54.606 INFO:teuthology.orchestra.run.smithi029.stdout:ceph noarch packages 31 kB/s | 9.5 kB 00:00 2024-01-24T13:25:54.683 INFO:teuthology.orchestra.run.smithi163.stdout:ceph noarch packages 30 kB/s | 9.5 kB 00:00 2024-01-24T13:25:54.791 INFO:teuthology.orchestra.run.smithi187.stdout:ceph source packages 4.5 kB/s | 1.5 kB 00:00 2024-01-24T13:25:54.909 INFO:teuthology.orchestra.run.smithi029.stdout:ceph source packages 5.5 kB/s | 1.5 kB 00:00 2024-01-24T13:25:54.942 INFO:teuthology.orchestra.run.smithi163.stdout:ceph source packages 6.4 kB/s | 1.5 kB 00:00 2024-01-24T13:25:54.975 INFO:teuthology.orchestra.run.smithi187.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2024-01-24T13:25:55.135 INFO:teuthology.orchestra.run.smithi029.stdout:Copr repo for python3-asyncssh owned by ceph 17 kB/s | 3.5 kB 00:00 2024-01-24T13:25:55.159 INFO:teuthology.orchestra.run.smithi163.stdout:Copr repo for python3-asyncssh owned by ceph 18 kB/s | 3.5 kB 00:00 2024-01-24T13:25:56.492 INFO:teuthology.orchestra.run.smithi187.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2024-01-24T13:25:56.570 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2024-01-24T13:25:57.271 INFO:teuthology.orchestra.run.smithi163.stdout:Extra Packages for Enterprise Linux 7.7 MB/s | 16 MB 00:02 2024-01-24T13:26:00.059 INFO:teuthology.orchestra.run.smithi187.stdout:lab-extras 428 kB/s | 24 kB 00:00 2024-01-24T13:26:00.581 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 438 kB/s | 24 kB 00:00 2024-01-24T13:26:00.878 INFO:teuthology.orchestra.run.smithi163.stdout:lab-extras 265 kB/s | 24 kB 00:00 2024-01-24T13:26:01.311 INFO:teuthology.orchestra.run.smithi187.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 50 MB/s | 59 MB 00:01 2024-01-24T13:26:02.577 INFO:teuthology.orchestra.run.smithi029.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 31 MB/s | 59 MB 00:01 2024-01-24T13:26:03.006 INFO:teuthology.orchestra.run.smithi163.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 29 MB/s | 59 MB 00:02 2024-01-24T13:26:12.595 INFO:teuthology.orchestra.run.smithi187.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 50 MB/s | 65 MB 00:01 2024-01-24T13:26:13.464 INFO:teuthology.orchestra.run.smithi163.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 14 MB/s | 8.8 MB 00:00 2024-01-24T13:26:14.146 INFO:teuthology.orchestra.run.smithi029.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 14 MB/s | 8.8 MB 00:00 2024-01-24T13:26:16.357 INFO:teuthology.orchestra.run.smithi163.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 51 MB/s | 65 MB 00:01 2024-01-24T13:26:17.725 INFO:teuthology.orchestra.run.smithi029.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 36 MB/s | 65 MB 00:01 2024-01-24T13:26:22.894 INFO:teuthology.orchestra.run.smithi187.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 14 MB/s | 8.8 MB 00:00 2024-01-24T13:26:30.084 INFO:teuthology.orchestra.run.smithi187.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-01-24T13:26:30.085 INFO:teuthology.orchestra.run.smithi187.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-01-24T13:26:30.103 INFO:teuthology.orchestra.run.smithi187.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2024-01-24T13:26:30.104 INFO:teuthology.orchestra.run.smithi187.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2024-01-24T13:26:30.106 INFO:teuthology.orchestra.run.smithi187.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2024-01-24T13:26:30.107 INFO:teuthology.orchestra.run.smithi187.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-01-24T13:26:30.110 INFO:teuthology.orchestra.run.smithi187.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-01-24T13:26:30.114 INFO:teuthology.orchestra.run.smithi187.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-01-24T13:26:30.115 INFO:teuthology.orchestra.run.smithi187.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-01-24T13:26:30.329 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-01-24T13:26:30.343 INFO:teuthology.orchestra.run.smithi187.stdout:=========================================================================================================================== 2024-01-24T13:26:30.343 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-01-24T13:26:30.343 INFO:teuthology.orchestra.run.smithi187.stdout:=========================================================================================================================== 2024-01-24T13:26:30.343 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-01-24T13:26:30.343 INFO:teuthology.orchestra.run.smithi187.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-13.el8_9 rhel-8-for-x86_64-baseos-rpms 144 k 2024-01-24T13:26:30.343 INFO:teuthology.orchestra.run.smithi187.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2024-01-24T13:26:30.344 INFO:teuthology.orchestra.run.smithi187.stdout: ceph x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 6.4 k 2024-01-24T13:26:30.344 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-base x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 5.4 M 2024-01-24T13:26:30.344 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-fuse x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 852 k 2024-01-24T13:26:30.344 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-immutable-object-cache x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 165 k 2024-01-24T13:26:30.344 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 1.6 M 2024-01-24T13:26:30.344 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-cephadm noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 132 k 2024-01-24T13:26:30.347 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-dashboard noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 1.8 M 2024-01-24T13:26:30.347 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 7.4 M 2024-01-24T13:26:30.347 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-rook noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 50 k 2024-01-24T13:26:30.347 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-radosgw x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 13 M 2024-01-24T13:26:30.347 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-test x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 47 M 2024-01-24T13:26:30.347 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-volume noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 262 k 2024-01-24T13:26:30.348 INFO:teuthology.orchestra.run.smithi187.stdout: cephadm noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 171 k 2024-01-24T13:26:30.348 INFO:teuthology.orchestra.run.smithi187.stdout: cephfs-mirror x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 259 k 2024-01-24T13:26:30.348 INFO:teuthology.orchestra.run.smithi187.stdout: cephfs-top noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 18 k 2024-01-24T13:26:30.348 INFO:teuthology.orchestra.run.smithi187.stdout: elfutils-libelf-devel x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 62 k 2024-01-24T13:26:30.348 INFO:teuthology.orchestra.run.smithi187.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2024-01-24T13:26:30.348 INFO:teuthology.orchestra.run.smithi187.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2024-01-24T13:26:30.348 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs-devel x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 31 k 2024-01-24T13:26:30.348 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs2 x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 776 k 2024-01-24T13:26:30.348 INFO:teuthology.orchestra.run.smithi187.stdout: librados-devel x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 139 k 2024-01-24T13:26:30.348 INFO:teuthology.orchestra.run.smithi187.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2024-01-24T13:26:30.348 INFO:teuthology.orchestra.run.smithi187.stdout: libuuid-devel x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 100 k 2024-01-24T13:26:30.349 INFO:teuthology.orchestra.run.smithi187.stdout: openssl-devel x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 2.3 M 2024-01-24T13:26:30.349 INFO:teuthology.orchestra.run.smithi187.stdout: postgresql x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 1.5 M 2024-01-24T13:26:30.349 INFO:teuthology.orchestra.run.smithi187.stdout: postgresql-contrib x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 811 k 2024-01-24T13:26:30.349 INFO:teuthology.orchestra.run.smithi187.stdout: postgresql-server x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 5.1 M 2024-01-24T13:26:30.349 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cephfs x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 197 k 2024-01-24T13:26:30.349 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rados x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 371 k 2024-01-24T13:26:30.349 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rbd x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 365 k 2024-01-24T13:26:30.349 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rgw x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 107 k 2024-01-24T13:26:30.349 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-fuse x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 94 k 2024-01-24T13:26:30.349 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-mirror x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 3.4 M 2024-01-24T13:26:30.349 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-nbd x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 181 k 2024-01-24T13:26:30.350 INFO:teuthology.orchestra.run.smithi187.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 73 k 2024-01-24T13:26:30.350 INFO:teuthology.orchestra.run.smithi187.stdout:Upgrading: 2024-01-24T13:26:30.350 INFO:teuthology.orchestra.run.smithi187.stdout: elfutils-libelf x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 232 k 2024-01-24T13:26:30.350 INFO:teuthology.orchestra.run.smithi187.stdout: elfutils-libs x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 303 k 2024-01-24T13:26:30.350 INFO:teuthology.orchestra.run.smithi187.stdout: iproute x86_64 6.2.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 854 k 2024-01-24T13:26:30.350 INFO:teuthology.orchestra.run.smithi187.stdout: libblkid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-01-24T13:26:30.350 INFO:teuthology.orchestra.run.smithi187.stdout: libfdisk x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 254 k 2024-01-24T13:26:30.350 INFO:teuthology.orchestra.run.smithi187.stdout: libmount x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 237 k 2024-01-24T13:26:30.350 INFO:teuthology.orchestra.run.smithi187.stdout: librados2 x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 3.7 M 2024-01-24T13:26:30.350 INFO:teuthology.orchestra.run.smithi187.stdout: librbd1 x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 3.6 M 2024-01-24T13:26:30.351 INFO:teuthology.orchestra.run.smithi187.stdout: libsmartcols x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 179 k 2024-01-24T13:26:30.351 INFO:teuthology.orchestra.run.smithi187.stdout: libuuid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 99 k 2024-01-24T13:26:30.351 INFO:teuthology.orchestra.run.smithi187.stdout: openssl x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 711 k 2024-01-24T13:26:30.351 INFO:teuthology.orchestra.run.smithi187.stdout: openssl-libs x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 1.5 M 2024-01-24T13:26:30.351 INFO:teuthology.orchestra.run.smithi187.stdout: util-linux x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 2.5 M 2024-01-24T13:26:30.351 INFO:teuthology.orchestra.run.smithi187.stdout: util-linux-user x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 103 k 2024-01-24T13:26:30.351 INFO:teuthology.orchestra.run.smithi187.stdout: xfsdump x86_64 3.1.8-7.el8_9 rhel-8-for-x86_64-baseos-rpms 333 k 2024-01-24T13:26:30.351 INFO:teuthology.orchestra.run.smithi187.stdout: xfsprogs x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M 2024-01-24T13:26:30.351 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2024-01-24T13:26:30.351 INFO:teuthology.orchestra.run.smithi187.stdout: automake noarch 1.16.1-8.el8 rhel-8-for-x86_64-appstream-rpms 714 k 2024-01-24T13:26:30.351 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-common x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 22 M 2024-01-24T13:26:30.351 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-grafana-dashboards noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 23 k 2024-01-24T13:26:30.352 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mds x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 2.3 M 2024-01-24T13:26:30.352 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-modules-core noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 242 k 2024-01-24T13:26:30.352 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mon x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 4.6 M 2024-01-24T13:26:30.352 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-osd x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 18 M 2024-01-24T13:26:30.352 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-prometheus-alerts noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 15 k 2024-01-24T13:26:30.352 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-selinux x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 24 k 2024-01-24T13:26:30.352 INFO:teuthology.orchestra.run.smithi187.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-01-24T13:26:30.352 INFO:teuthology.orchestra.run.smithi187.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-24T13:26:30.352 INFO:teuthology.orchestra.run.smithi187.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-01-24T13:26:30.352 INFO:teuthology.orchestra.run.smithi187.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2024-01-24T13:26:30.352 INFO:teuthology.orchestra.run.smithi187.stdout: krb5-devel x86_64 1.18.2-26.el8_9 rhel-8-for-x86_64-baseos-rpms 562 k 2024-01-24T13:26:30.353 INFO:teuthology.orchestra.run.smithi187.stdout: libcephsqlite x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 141 k 2024-01-24T13:26:30.353 INFO:teuthology.orchestra.run.smithi187.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2024-01-24T13:26:30.353 INFO:teuthology.orchestra.run.smithi187.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-01-24T13:26:30.353 INFO:teuthology.orchestra.run.smithi187.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-24T13:26:30.353 INFO:teuthology.orchestra.run.smithi187.stdout: libpq x86_64 13.11-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2024-01-24T13:26:30.353 INFO:teuthology.orchestra.run.smithi187.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-01-24T13:26:30.353 INFO:teuthology.orchestra.run.smithi187.stdout: librabbitmq x86_64 0.9.0-4.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2024-01-24T13:26:30.353 INFO:teuthology.orchestra.run.smithi187.stdout: libradosstriper1 x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 474 k 2024-01-24T13:26:30.353 INFO:teuthology.orchestra.run.smithi187.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-01-24T13:26:30.353 INFO:teuthology.orchestra.run.smithi187.stdout: librgw2 x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 6.5 M 2024-01-24T13:26:30.353 INFO:teuthology.orchestra.run.smithi187.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2024-01-24T13:26:30.353 INFO:teuthology.orchestra.run.smithi187.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2024-01-24T13:26:30.354 INFO:teuthology.orchestra.run.smithi187.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-24T13:26:30.354 INFO:teuthology.orchestra.run.smithi187.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2024-01-24T13:26:30.354 INFO:teuthology.orchestra.run.smithi187.stdout: libzstd-devel x86_64 1.4.4-1.el8 rhel-8-for-x86_64-baseos-rpms 44 k 2024-01-24T13:26:30.354 INFO:teuthology.orchestra.run.smithi187.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-01-24T13:26:30.354 INFO:teuthology.orchestra.run.smithi187.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2024-01-24T13:26:30.354 INFO:teuthology.orchestra.run.smithi187.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-01-24T13:26:30.354 INFO:teuthology.orchestra.run.smithi187.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-01-24T13:26:30.354 INFO:teuthology.orchestra.run.smithi187.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2024-01-24T13:26:30.354 INFO:teuthology.orchestra.run.smithi187.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2024-01-24T13:26:30.354 INFO:teuthology.orchestra.run.smithi187.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-01-24T13:26:30.354 INFO:teuthology.orchestra.run.smithi187.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-01-24T13:26:30.354 INFO:teuthology.orchestra.run.smithi187.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-01-24T13:26:30.355 INFO:teuthology.orchestra.run.smithi187.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-24T13:26:30.355 INFO:teuthology.orchestra.run.smithi187.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-24T13:26:30.355 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-24T13:26:30.355 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-argparse x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 45 k 2024-01-24T13:26:30.355 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-common x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 123 k 2024-01-24T13:26:30.355 INFO:teuthology.orchestra.run.smithi187.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-24T13:26:30.355 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-01-24T13:26:30.355 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-24T13:26:30.355 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-24T13:26:30.355 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cryptography x86_64 3.2.1-6.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2024-01-24T13:26:30.355 INFO:teuthology.orchestra.run.smithi187.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-24T13:26:30.355 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-24T13:26:30.356 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-24T13:26:30.356 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-01-24T13:26:30.356 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-01-24T13:26:30.356 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-01-24T13:26:30.356 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-01-24T13:26:30.356 INFO:teuthology.orchestra.run.smithi187.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-24T13:26:30.356 INFO:teuthology.orchestra.run.smithi187.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-24T13:26:30.356 INFO:teuthology.orchestra.run.smithi187.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-01-24T13:26:30.356 INFO:teuthology.orchestra.run.smithi187.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-01-24T13:26:30.356 INFO:teuthology.orchestra.run.smithi187.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-24T13:26:30.356 INFO:teuthology.orchestra.run.smithi187.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-01-24T13:26:30.357 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-01-24T13:26:30.357 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-01-24T13:26:30.357 INFO:teuthology.orchestra.run.smithi187.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-01-24T13:26:30.357 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-24T13:26:30.357 INFO:teuthology.orchestra.run.smithi187.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-24T13:26:30.357 INFO:teuthology.orchestra.run.smithi187.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-01-24T13:26:30.357 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-01-24T13:26:30.357 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-01-24T13:26:30.357 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-01-24T13:26:30.357 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-01-24T13:26:30.357 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-01-24T13:26:30.357 INFO:teuthology.orchestra.run.smithi187.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-24T13:26:30.358 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-01-24T13:26:30.358 INFO:teuthology.orchestra.run.smithi187.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-24T13:26:30.358 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-24T13:26:30.358 INFO:teuthology.orchestra.run.smithi187.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-01-24T13:26:30.358 INFO:teuthology.orchestra.run.smithi187.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-24T13:26:30.358 INFO:teuthology.orchestra.run.smithi187.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-24T13:26:30.358 INFO:teuthology.orchestra.run.smithi187.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-24T13:26:30.358 INFO:teuthology.orchestra.run.smithi187.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-24T13:26:30.358 INFO:teuthology.orchestra.run.smithi187.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-24T13:26:30.358 INFO:teuthology.orchestra.run.smithi187.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-24T13:26:30.358 INFO:teuthology.orchestra.run.smithi187.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-24T13:26:30.358 INFO:teuthology.orchestra.run.smithi187.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-24T13:26:30.359 INFO:teuthology.orchestra.run.smithi187.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-01-24T13:26:30.359 INFO:teuthology.orchestra.run.smithi187.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-24T13:26:30.359 INFO:teuthology.orchestra.run.smithi187.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-01-24T13:26:30.359 INFO:teuthology.orchestra.run.smithi187.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-01-24T13:26:30.359 INFO:teuthology.orchestra.run.smithi187.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2024-01-24T13:26:30.359 INFO:teuthology.orchestra.run.smithi187.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-01-24T13:26:30.359 INFO:teuthology.orchestra.run.smithi187.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2024-01-24T13:26:30.359 INFO:teuthology.orchestra.run.smithi187.stdout:Enabling module streams: 2024-01-24T13:26:30.359 INFO:teuthology.orchestra.run.smithi187.stdout: postgresql 10 2024-01-24T13:26:30.359 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-01-24T13:26:30.359 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-01-24T13:26:30.359 INFO:teuthology.orchestra.run.smithi187.stdout:=========================================================================================================================== 2024-01-24T13:26:30.360 INFO:teuthology.orchestra.run.smithi187.stdout:Install 129 Packages 2024-01-24T13:26:30.360 INFO:teuthology.orchestra.run.smithi187.stdout:Upgrade 16 Packages 2024-01-24T13:26:30.360 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-01-24T13:26:30.361 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 212 M 2024-01-24T13:26:30.361 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-01-24T13:26:30.570 INFO:teuthology.orchestra.run.smithi187.stdout:(1/145): ceph-18.2.1-341.g4e3182a6.el8.x86_64.r 36 kB/s | 6.4 kB 00:00 2024-01-24T13:26:30.912 INFO:teuthology.orchestra.run.smithi187.stdout:(2/145): ceph-fuse-18.2.1-341.g4e3182a6.el8.x86 2.4 MB/s | 852 kB 00:00 2024-01-24T13:26:31.012 INFO:teuthology.orchestra.run.smithi187.stdout:(3/145): ceph-immutable-object-cache-18.2.1-341 1.6 MB/s | 165 kB 00:00 2024-01-24T13:26:31.104 INFO:teuthology.orchestra.run.smithi163.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-01-24T13:26:31.105 INFO:teuthology.orchestra.run.smithi163.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-01-24T13:26:31.121 INFO:teuthology.orchestra.run.smithi187.stdout:(4/145): ceph-base-18.2.1-341.g4e3182a6.el8.x86 7.4 MB/s | 5.4 MB 00:00 2024-01-24T13:26:31.123 INFO:teuthology.orchestra.run.smithi163.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2024-01-24T13:26:31.124 INFO:teuthology.orchestra.run.smithi163.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2024-01-24T13:26:31.125 INFO:teuthology.orchestra.run.smithi163.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2024-01-24T13:26:31.127 INFO:teuthology.orchestra.run.smithi163.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-01-24T13:26:31.129 INFO:teuthology.orchestra.run.smithi163.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-01-24T13:26:31.133 INFO:teuthology.orchestra.run.smithi163.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-01-24T13:26:31.135 INFO:teuthology.orchestra.run.smithi163.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-01-24T13:26:31.313 INFO:teuthology.orchestra.run.smithi187.stdout:(5/145): ceph-mgr-18.2.1-341.g4e3182a6.el8.x86_ 8.1 MB/s | 1.6 MB 00:00 2024-01-24T13:26:31.340 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-01-24T13:26:31.354 INFO:teuthology.orchestra.run.smithi163.stdout:=========================================================================================================================== 2024-01-24T13:26:31.354 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-01-24T13:26:31.354 INFO:teuthology.orchestra.run.smithi163.stdout:=========================================================================================================================== 2024-01-24T13:26:31.354 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-01-24T13:26:31.354 INFO:teuthology.orchestra.run.smithi163.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-13.el8_9 rhel-8-for-x86_64-baseos-rpms 144 k 2024-01-24T13:26:31.354 INFO:teuthology.orchestra.run.smithi163.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2024-01-24T13:26:31.355 INFO:teuthology.orchestra.run.smithi163.stdout: ceph x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 6.4 k 2024-01-24T13:26:31.355 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-base x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 5.4 M 2024-01-24T13:26:31.355 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-fuse x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 852 k 2024-01-24T13:26:31.355 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-immutable-object-cache x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 165 k 2024-01-24T13:26:31.355 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 1.6 M 2024-01-24T13:26:31.355 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-cephadm noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 132 k 2024-01-24T13:26:31.355 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-dashboard noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 1.8 M 2024-01-24T13:26:31.355 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 7.4 M 2024-01-24T13:26:31.355 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-rook noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 50 k 2024-01-24T13:26:31.355 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-radosgw x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 13 M 2024-01-24T13:26:31.355 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-test x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 47 M 2024-01-24T13:26:31.355 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-volume noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 262 k 2024-01-24T13:26:31.355 INFO:teuthology.orchestra.run.smithi163.stdout: cephadm noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 171 k 2024-01-24T13:26:31.356 INFO:teuthology.orchestra.run.smithi163.stdout: cephfs-mirror x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 259 k 2024-01-24T13:26:31.356 INFO:teuthology.orchestra.run.smithi163.stdout: cephfs-top noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 18 k 2024-01-24T13:26:31.356 INFO:teuthology.orchestra.run.smithi163.stdout: elfutils-libelf-devel x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 62 k 2024-01-24T13:26:31.356 INFO:teuthology.orchestra.run.smithi163.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2024-01-24T13:26:31.356 INFO:teuthology.orchestra.run.smithi163.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2024-01-24T13:26:31.356 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs-devel x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 31 k 2024-01-24T13:26:31.356 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs2 x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 776 k 2024-01-24T13:26:31.356 INFO:teuthology.orchestra.run.smithi163.stdout: librados-devel x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 139 k 2024-01-24T13:26:31.356 INFO:teuthology.orchestra.run.smithi163.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2024-01-24T13:26:31.356 INFO:teuthology.orchestra.run.smithi163.stdout: libuuid-devel x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 100 k 2024-01-24T13:26:31.356 INFO:teuthology.orchestra.run.smithi163.stdout: openssl-devel x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 2.3 M 2024-01-24T13:26:31.356 INFO:teuthology.orchestra.run.smithi163.stdout: postgresql x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 1.5 M 2024-01-24T13:26:31.357 INFO:teuthology.orchestra.run.smithi163.stdout: postgresql-contrib x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 811 k 2024-01-24T13:26:31.357 INFO:teuthology.orchestra.run.smithi163.stdout: postgresql-server x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 5.1 M 2024-01-24T13:26:31.357 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cephfs x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 197 k 2024-01-24T13:26:31.357 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rados x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 371 k 2024-01-24T13:26:31.357 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rbd x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 365 k 2024-01-24T13:26:31.357 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rgw x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 107 k 2024-01-24T13:26:31.357 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-fuse x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 94 k 2024-01-24T13:26:31.357 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-mirror x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 3.4 M 2024-01-24T13:26:31.357 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-nbd x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 181 k 2024-01-24T13:26:31.357 INFO:teuthology.orchestra.run.smithi163.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 73 k 2024-01-24T13:26:31.357 INFO:teuthology.orchestra.run.smithi163.stdout:Upgrading: 2024-01-24T13:26:31.357 INFO:teuthology.orchestra.run.smithi163.stdout: elfutils-libelf x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 232 k 2024-01-24T13:26:31.357 INFO:teuthology.orchestra.run.smithi163.stdout: elfutils-libs x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 303 k 2024-01-24T13:26:31.357 INFO:teuthology.orchestra.run.smithi163.stdout: iproute x86_64 6.2.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 854 k 2024-01-24T13:26:31.357 INFO:teuthology.orchestra.run.smithi163.stdout: libblkid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-01-24T13:26:31.358 INFO:teuthology.orchestra.run.smithi163.stdout: libfdisk x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 254 k 2024-01-24T13:26:31.358 INFO:teuthology.orchestra.run.smithi163.stdout: libmount x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 237 k 2024-01-24T13:26:31.358 INFO:teuthology.orchestra.run.smithi163.stdout: librados2 x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 3.7 M 2024-01-24T13:26:31.358 INFO:teuthology.orchestra.run.smithi163.stdout: librbd1 x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 3.6 M 2024-01-24T13:26:31.358 INFO:teuthology.orchestra.run.smithi163.stdout: libsmartcols x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 179 k 2024-01-24T13:26:31.358 INFO:teuthology.orchestra.run.smithi163.stdout: libuuid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 99 k 2024-01-24T13:26:31.358 INFO:teuthology.orchestra.run.smithi163.stdout: openssl x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 711 k 2024-01-24T13:26:31.358 INFO:teuthology.orchestra.run.smithi163.stdout: openssl-libs x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 1.5 M 2024-01-24T13:26:31.358 INFO:teuthology.orchestra.run.smithi163.stdout: util-linux x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 2.5 M 2024-01-24T13:26:31.358 INFO:teuthology.orchestra.run.smithi163.stdout: util-linux-user x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 103 k 2024-01-24T13:26:31.358 INFO:teuthology.orchestra.run.smithi163.stdout: xfsdump x86_64 3.1.8-7.el8_9 rhel-8-for-x86_64-baseos-rpms 333 k 2024-01-24T13:26:31.358 INFO:teuthology.orchestra.run.smithi163.stdout: xfsprogs x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M 2024-01-24T13:26:31.358 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-01-24T13:26:31.358 INFO:teuthology.orchestra.run.smithi163.stdout: automake noarch 1.16.1-8.el8 rhel-8-for-x86_64-appstream-rpms 714 k 2024-01-24T13:26:31.359 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-common x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 22 M 2024-01-24T13:26:31.359 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-grafana-dashboards noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 23 k 2024-01-24T13:26:31.359 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mds x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 2.3 M 2024-01-24T13:26:31.359 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-modules-core noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 242 k 2024-01-24T13:26:31.359 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mon x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 4.6 M 2024-01-24T13:26:31.359 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-osd x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 18 M 2024-01-24T13:26:31.359 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-prometheus-alerts noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 15 k 2024-01-24T13:26:31.359 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-selinux x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 24 k 2024-01-24T13:26:31.359 INFO:teuthology.orchestra.run.smithi163.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-01-24T13:26:31.359 INFO:teuthology.orchestra.run.smithi163.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-24T13:26:31.359 INFO:teuthology.orchestra.run.smithi163.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-01-24T13:26:31.359 INFO:teuthology.orchestra.run.smithi163.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2024-01-24T13:26:31.359 INFO:teuthology.orchestra.run.smithi163.stdout: krb5-devel x86_64 1.18.2-26.el8_9 rhel-8-for-x86_64-baseos-rpms 562 k 2024-01-24T13:26:31.359 INFO:teuthology.orchestra.run.smithi163.stdout: libcephsqlite x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 141 k 2024-01-24T13:26:31.359 INFO:teuthology.orchestra.run.smithi163.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2024-01-24T13:26:31.360 INFO:teuthology.orchestra.run.smithi163.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-01-24T13:26:31.360 INFO:teuthology.orchestra.run.smithi163.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-24T13:26:31.360 INFO:teuthology.orchestra.run.smithi163.stdout: libpq x86_64 13.11-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2024-01-24T13:26:31.360 INFO:teuthology.orchestra.run.smithi163.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-01-24T13:26:31.360 INFO:teuthology.orchestra.run.smithi163.stdout: librabbitmq x86_64 0.9.0-4.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2024-01-24T13:26:31.360 INFO:teuthology.orchestra.run.smithi163.stdout: libradosstriper1 x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 474 k 2024-01-24T13:26:31.360 INFO:teuthology.orchestra.run.smithi163.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-01-24T13:26:31.360 INFO:teuthology.orchestra.run.smithi163.stdout: librgw2 x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 6.5 M 2024-01-24T13:26:31.360 INFO:teuthology.orchestra.run.smithi163.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2024-01-24T13:26:31.360 INFO:teuthology.orchestra.run.smithi163.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2024-01-24T13:26:31.360 INFO:teuthology.orchestra.run.smithi163.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-24T13:26:31.360 INFO:teuthology.orchestra.run.smithi163.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2024-01-24T13:26:31.360 INFO:teuthology.orchestra.run.smithi163.stdout: libzstd-devel x86_64 1.4.4-1.el8 rhel-8-for-x86_64-baseos-rpms 44 k 2024-01-24T13:26:31.361 INFO:teuthology.orchestra.run.smithi163.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-01-24T13:26:31.361 INFO:teuthology.orchestra.run.smithi163.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2024-01-24T13:26:31.361 INFO:teuthology.orchestra.run.smithi163.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-01-24T13:26:31.361 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-01-24T13:26:31.361 INFO:teuthology.orchestra.run.smithi163.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2024-01-24T13:26:31.361 INFO:teuthology.orchestra.run.smithi163.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2024-01-24T13:26:31.361 INFO:teuthology.orchestra.run.smithi163.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-01-24T13:26:31.361 INFO:teuthology.orchestra.run.smithi163.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-01-24T13:26:31.361 INFO:teuthology.orchestra.run.smithi163.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-01-24T13:26:31.361 INFO:teuthology.orchestra.run.smithi163.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-24T13:26:31.362 INFO:teuthology.orchestra.run.smithi163.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-24T13:26:31.362 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-24T13:26:31.362 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-argparse x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 45 k 2024-01-24T13:26:31.362 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-common x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 123 k 2024-01-24T13:26:31.362 INFO:teuthology.orchestra.run.smithi163.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-24T13:26:31.362 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-01-24T13:26:31.362 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-24T13:26:31.362 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-24T13:26:31.362 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cryptography x86_64 3.2.1-6.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2024-01-24T13:26:31.362 INFO:teuthology.orchestra.run.smithi163.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-24T13:26:31.362 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-24T13:26:31.363 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-24T13:26:31.363 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-01-24T13:26:31.363 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-01-24T13:26:31.363 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-01-24T13:26:31.363 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-01-24T13:26:31.363 INFO:teuthology.orchestra.run.smithi163.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-24T13:26:31.363 INFO:teuthology.orchestra.run.smithi163.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-24T13:26:31.363 INFO:teuthology.orchestra.run.smithi163.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-01-24T13:26:31.363 INFO:teuthology.orchestra.run.smithi163.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-01-24T13:26:31.363 INFO:teuthology.orchestra.run.smithi163.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-24T13:26:31.363 INFO:teuthology.orchestra.run.smithi163.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-01-24T13:26:31.363 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-01-24T13:26:31.363 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-01-24T13:26:31.363 INFO:teuthology.orchestra.run.smithi163.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-01-24T13:26:31.364 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-24T13:26:31.364 INFO:teuthology.orchestra.run.smithi163.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-24T13:26:31.364 INFO:teuthology.orchestra.run.smithi163.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-01-24T13:26:31.364 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-01-24T13:26:31.364 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-01-24T13:26:31.364 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-01-24T13:26:31.364 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-01-24T13:26:31.364 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-01-24T13:26:31.364 INFO:teuthology.orchestra.run.smithi163.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-24T13:26:31.364 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-01-24T13:26:31.364 INFO:teuthology.orchestra.run.smithi163.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-24T13:26:31.364 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-24T13:26:31.365 INFO:teuthology.orchestra.run.smithi163.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-01-24T13:26:31.365 INFO:teuthology.orchestra.run.smithi163.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-24T13:26:31.365 INFO:teuthology.orchestra.run.smithi163.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-24T13:26:31.365 INFO:teuthology.orchestra.run.smithi163.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-24T13:26:31.365 INFO:teuthology.orchestra.run.smithi163.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-24T13:26:31.365 INFO:teuthology.orchestra.run.smithi163.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-24T13:26:31.365 INFO:teuthology.orchestra.run.smithi163.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-24T13:26:31.365 INFO:teuthology.orchestra.run.smithi163.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-24T13:26:31.365 INFO:teuthology.orchestra.run.smithi163.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-24T13:26:31.365 INFO:teuthology.orchestra.run.smithi163.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-01-24T13:26:31.365 INFO:teuthology.orchestra.run.smithi163.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-24T13:26:31.365 INFO:teuthology.orchestra.run.smithi163.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-01-24T13:26:31.366 INFO:teuthology.orchestra.run.smithi163.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-01-24T13:26:31.366 INFO:teuthology.orchestra.run.smithi163.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2024-01-24T13:26:31.366 INFO:teuthology.orchestra.run.smithi163.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-01-24T13:26:31.366 INFO:teuthology.orchestra.run.smithi163.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2024-01-24T13:26:31.366 INFO:teuthology.orchestra.run.smithi163.stdout:Enabling module streams: 2024-01-24T13:26:31.366 INFO:teuthology.orchestra.run.smithi163.stdout: postgresql 10 2024-01-24T13:26:31.366 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-01-24T13:26:31.366 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-01-24T13:26:31.366 INFO:teuthology.orchestra.run.smithi163.stdout:=========================================================================================================================== 2024-01-24T13:26:31.366 INFO:teuthology.orchestra.run.smithi163.stdout:Install 129 Packages 2024-01-24T13:26:31.366 INFO:teuthology.orchestra.run.smithi163.stdout:Upgrade 16 Packages 2024-01-24T13:26:31.366 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-01-24T13:26:31.367 INFO:teuthology.orchestra.run.smithi187.stdout:(6/145): ceph-mds-18.2.1-341.g4e3182a6.el8.x86_ 6.5 MB/s | 2.3 MB 00:00 2024-01-24T13:26:31.367 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 212 M 2024-01-24T13:26:31.367 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-01-24T13:26:31.724 INFO:teuthology.orchestra.run.smithi163.stdout:(1/145): ceph-18.2.1-341.g4e3182a6.el8.x86_64.r 22 kB/s | 6.4 kB 00:00 2024-01-24T13:26:31.804 INFO:teuthology.orchestra.run.smithi187.stdout:(7/145): ceph-mon-18.2.1-341.g4e3182a6.el8.x86_ 9.4 MB/s | 4.6 MB 00:00 2024-01-24T13:26:32.366 INFO:teuthology.orchestra.run.smithi163.stdout:(2/145): ceph-fuse-18.2.1-341.g4e3182a6.el8.x86 1.3 MB/s | 852 kB 00:00 2024-01-24T13:26:32.533 INFO:teuthology.orchestra.run.smithi163.stdout:(3/145): ceph-immutable-object-cache-18.2.1-341 994 kB/s | 165 kB 00:00 2024-01-24T13:26:32.763 INFO:teuthology.orchestra.run.smithi187.stdout:(8/145): ceph-common-18.2.1-341.g4e3182a6.el8.x 9.1 MB/s | 22 MB 00:02 2024-01-24T13:26:32.847 INFO:teuthology.orchestra.run.smithi187.stdout:(9/145): ceph-selinux-18.2.1-341.g4e3182a6.el8. 292 kB/s | 24 kB 00:00 2024-01-24T13:26:33.431 INFO:teuthology.orchestra.run.smithi187.stdout:(10/145): ceph-radosgw-18.2.1-341.g4e3182a6.el8 8.0 MB/s | 13 MB 00:01 2024-01-24T13:26:33.533 INFO:teuthology.orchestra.run.smithi163.stdout:(4/145): ceph-base-18.2.1-341.g4e3182a6.el8.x86 2.6 MB/s | 5.4 MB 00:02 2024-01-24T13:26:33.539 INFO:teuthology.orchestra.run.smithi187.stdout:(11/145): cephfs-mirror-18.2.1-341.g4e3182a6.el 2.3 MB/s | 259 kB 00:00 2024-01-24T13:26:33.648 INFO:teuthology.orchestra.run.smithi187.stdout:(12/145): libcephfs-devel-18.2.1-341.g4e3182a6. 285 kB/s | 31 kB 00:00 2024-01-24T13:26:33.842 INFO:teuthology.orchestra.run.smithi163.stdout:(5/145): ceph-mds-18.2.1-341.g4e3182a6.el8.x86_ 1.8 MB/s | 2.3 MB 00:01 2024-01-24T13:26:34.073 INFO:teuthology.orchestra.run.smithi187.stdout:(13/145): libcephfs2-18.2.1-341.g4e3182a6.el8.x 1.8 MB/s | 776 kB 00:00 2024-01-24T13:26:34.138 INFO:teuthology.orchestra.run.smithi029.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-01-24T13:26:34.139 INFO:teuthology.orchestra.run.smithi029.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-01-24T13:26:34.160 INFO:teuthology.orchestra.run.smithi029.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2024-01-24T13:26:34.161 INFO:teuthology.orchestra.run.smithi029.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2024-01-24T13:26:34.163 INFO:teuthology.orchestra.run.smithi029.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2024-01-24T13:26:34.164 INFO:teuthology.orchestra.run.smithi029.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-01-24T13:26:34.167 INFO:teuthology.orchestra.run.smithi163.stdout:(6/145): ceph-mgr-18.2.1-341.g4e3182a6.el8.x86_ 2.5 MB/s | 1.6 MB 00:00 2024-01-24T13:26:34.167 INFO:teuthology.orchestra.run.smithi029.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-01-24T13:26:34.172 INFO:teuthology.orchestra.run.smithi029.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-01-24T13:26:34.174 INFO:teuthology.orchestra.run.smithi029.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-01-24T13:26:34.190 INFO:teuthology.orchestra.run.smithi187.stdout:(14/145): libcephsqlite-18.2.1-341.g4e3182a6.el 1.2 MB/s | 141 kB 00:00 2024-01-24T13:26:34.348 INFO:teuthology.orchestra.run.smithi187.stdout:(15/145): librados-devel-18.2.1-341.g4e3182a6.e 877 kB/s | 139 kB 00:00 2024-01-24T13:26:34.415 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-24T13:26:34.431 INFO:teuthology.orchestra.run.smithi029.stdout:=========================================================================================================================== 2024-01-24T13:26:34.431 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-01-24T13:26:34.431 INFO:teuthology.orchestra.run.smithi029.stdout:=========================================================================================================================== 2024-01-24T13:26:34.431 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-01-24T13:26:34.431 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-13.el8_9 rhel-8-for-x86_64-baseos-rpms 144 k 2024-01-24T13:26:34.431 INFO:teuthology.orchestra.run.smithi029.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2024-01-24T13:26:34.431 INFO:teuthology.orchestra.run.smithi029.stdout: ceph x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 6.4 k 2024-01-24T13:26:34.432 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 5.4 M 2024-01-24T13:26:34.432 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 852 k 2024-01-24T13:26:34.432 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 165 k 2024-01-24T13:26:34.432 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 1.6 M 2024-01-24T13:26:34.432 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 132 k 2024-01-24T13:26:34.432 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 1.8 M 2024-01-24T13:26:34.432 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 7.4 M 2024-01-24T13:26:34.432 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 50 k 2024-01-24T13:26:34.432 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 13 M 2024-01-24T13:26:34.432 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 47 M 2024-01-24T13:26:34.432 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-volume noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 262 k 2024-01-24T13:26:34.432 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 171 k 2024-01-24T13:26:34.432 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-mirror x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 259 k 2024-01-24T13:26:34.432 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-top noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 18 k 2024-01-24T13:26:34.433 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-libelf-devel x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 62 k 2024-01-24T13:26:34.433 INFO:teuthology.orchestra.run.smithi029.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2024-01-24T13:26:34.433 INFO:teuthology.orchestra.run.smithi029.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2024-01-24T13:26:34.433 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 31 k 2024-01-24T13:26:34.433 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2 x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 776 k 2024-01-24T13:26:34.433 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 139 k 2024-01-24T13:26:34.433 INFO:teuthology.orchestra.run.smithi029.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2024-01-24T13:26:34.433 INFO:teuthology.orchestra.run.smithi029.stdout: libuuid-devel x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 100 k 2024-01-24T13:26:34.433 INFO:teuthology.orchestra.run.smithi029.stdout: openssl-devel x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 2.3 M 2024-01-24T13:26:34.433 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 1.5 M 2024-01-24T13:26:34.433 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-contrib x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 811 k 2024-01-24T13:26:34.433 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-server x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 5.1 M 2024-01-24T13:26:34.433 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 197 k 2024-01-24T13:26:34.433 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 371 k 2024-01-24T13:26:34.434 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 365 k 2024-01-24T13:26:34.434 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 107 k 2024-01-24T13:26:34.434 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 94 k 2024-01-24T13:26:34.434 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 3.4 M 2024-01-24T13:26:34.434 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 181 k 2024-01-24T13:26:34.434 INFO:teuthology.orchestra.run.smithi029.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 73 k 2024-01-24T13:26:34.434 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrading: 2024-01-24T13:26:34.434 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-libelf x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 232 k 2024-01-24T13:26:34.434 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-libs x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 303 k 2024-01-24T13:26:34.434 INFO:teuthology.orchestra.run.smithi029.stdout: iproute x86_64 6.2.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 854 k 2024-01-24T13:26:34.434 INFO:teuthology.orchestra.run.smithi029.stdout: libblkid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-01-24T13:26:34.434 INFO:teuthology.orchestra.run.smithi029.stdout: libfdisk x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 254 k 2024-01-24T13:26:34.434 INFO:teuthology.orchestra.run.smithi029.stdout: libmount x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 237 k 2024-01-24T13:26:34.434 INFO:teuthology.orchestra.run.smithi029.stdout: librados2 x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 3.7 M 2024-01-24T13:26:34.435 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1 x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 3.6 M 2024-01-24T13:26:34.435 INFO:teuthology.orchestra.run.smithi029.stdout: libsmartcols x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 179 k 2024-01-24T13:26:34.435 INFO:teuthology.orchestra.run.smithi029.stdout: libuuid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 99 k 2024-01-24T13:26:34.435 INFO:teuthology.orchestra.run.smithi029.stdout: openssl x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 711 k 2024-01-24T13:26:34.435 INFO:teuthology.orchestra.run.smithi029.stdout: openssl-libs x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 1.5 M 2024-01-24T13:26:34.435 INFO:teuthology.orchestra.run.smithi029.stdout: util-linux x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 2.5 M 2024-01-24T13:26:34.435 INFO:teuthology.orchestra.run.smithi029.stdout: util-linux-user x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 103 k 2024-01-24T13:26:34.435 INFO:teuthology.orchestra.run.smithi029.stdout: xfsdump x86_64 3.1.8-7.el8_9 rhel-8-for-x86_64-baseos-rpms 333 k 2024-01-24T13:26:34.435 INFO:teuthology.orchestra.run.smithi029.stdout: xfsprogs x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M 2024-01-24T13:26:34.435 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-01-24T13:26:34.435 INFO:teuthology.orchestra.run.smithi029.stdout: automake noarch 1.16.1-8.el8 rhel-8-for-x86_64-appstream-rpms 714 k 2024-01-24T13:26:34.435 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 22 M 2024-01-24T13:26:34.435 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 23 k 2024-01-24T13:26:34.436 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 2.3 M 2024-01-24T13:26:34.436 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 242 k 2024-01-24T13:26:34.436 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 4.6 M 2024-01-24T13:26:34.436 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 18 M 2024-01-24T13:26:34.436 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 15 k 2024-01-24T13:26:34.436 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 24 k 2024-01-24T13:26:34.436 INFO:teuthology.orchestra.run.smithi029.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-01-24T13:26:34.436 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-24T13:26:34.436 INFO:teuthology.orchestra.run.smithi029.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-01-24T13:26:34.436 INFO:teuthology.orchestra.run.smithi029.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2024-01-24T13:26:34.436 INFO:teuthology.orchestra.run.smithi029.stdout: krb5-devel x86_64 1.18.2-26.el8_9 rhel-8-for-x86_64-baseos-rpms 562 k 2024-01-24T13:26:34.436 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 141 k 2024-01-24T13:26:34.436 INFO:teuthology.orchestra.run.smithi029.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2024-01-24T13:26:34.436 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-01-24T13:26:34.437 INFO:teuthology.orchestra.run.smithi029.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-24T13:26:34.437 INFO:teuthology.orchestra.run.smithi029.stdout: libpq x86_64 13.11-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2024-01-24T13:26:34.437 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-01-24T13:26:34.437 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq x86_64 0.9.0-4.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2024-01-24T13:26:34.437 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1 x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 474 k 2024-01-24T13:26:34.437 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-01-24T13:26:34.437 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2 x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 6.5 M 2024-01-24T13:26:34.437 INFO:teuthology.orchestra.run.smithi029.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2024-01-24T13:26:34.437 INFO:teuthology.orchestra.run.smithi029.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2024-01-24T13:26:34.437 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-24T13:26:34.437 INFO:teuthology.orchestra.run.smithi029.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2024-01-24T13:26:34.437 INFO:teuthology.orchestra.run.smithi029.stdout: libzstd-devel x86_64 1.4.4-1.el8 rhel-8-for-x86_64-baseos-rpms 44 k 2024-01-24T13:26:34.437 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-01-24T13:26:34.438 INFO:teuthology.orchestra.run.smithi029.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2024-01-24T13:26:34.438 INFO:teuthology.orchestra.run.smithi029.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-01-24T13:26:34.438 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-01-24T13:26:34.438 INFO:teuthology.orchestra.run.smithi029.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2024-01-24T13:26:34.438 INFO:teuthology.orchestra.run.smithi029.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2024-01-24T13:26:34.438 INFO:teuthology.orchestra.run.smithi029.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-01-24T13:26:34.438 INFO:teuthology.orchestra.run.smithi029.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-01-24T13:26:34.438 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-01-24T13:26:34.438 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-24T13:26:34.438 INFO:teuthology.orchestra.run.smithi029.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-24T13:26:34.438 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-24T13:26:34.438 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 45 k 2024-01-24T13:26:34.438 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 123 k 2024-01-24T13:26:34.438 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-24T13:26:34.438 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-01-24T13:26:34.439 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-24T13:26:34.439 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-24T13:26:34.439 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography x86_64 3.2.1-6.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2024-01-24T13:26:34.439 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-24T13:26:34.439 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-24T13:26:34.439 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-24T13:26:34.439 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-01-24T13:26:34.439 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-01-24T13:26:34.439 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-01-24T13:26:34.439 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-01-24T13:26:34.439 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-24T13:26:34.439 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-24T13:26:34.439 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-01-24T13:26:34.439 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-01-24T13:26:34.440 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-24T13:26:34.440 INFO:teuthology.orchestra.run.smithi029.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-01-24T13:26:34.440 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-01-24T13:26:34.440 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-01-24T13:26:34.440 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-01-24T13:26:34.440 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-24T13:26:34.440 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-24T13:26:34.440 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-01-24T13:26:34.440 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-01-24T13:26:34.440 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-01-24T13:26:34.440 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-01-24T13:26:34.440 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-01-24T13:26:34.441 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-01-24T13:26:34.441 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-24T13:26:34.441 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-01-24T13:26:34.441 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-24T13:26:34.441 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-24T13:26:34.441 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-01-24T13:26:34.441 INFO:teuthology.orchestra.run.smithi029.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-24T13:26:34.441 INFO:teuthology.orchestra.run.smithi029.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-24T13:26:34.441 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-24T13:26:34.441 INFO:teuthology.orchestra.run.smithi029.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-24T13:26:34.441 INFO:teuthology.orchestra.run.smithi029.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-24T13:26:34.441 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-24T13:26:34.441 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-24T13:26:34.441 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-24T13:26:34.442 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-01-24T13:26:34.442 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-24T13:26:34.442 INFO:teuthology.orchestra.run.smithi029.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-01-24T13:26:34.442 INFO:teuthology.orchestra.run.smithi029.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-01-24T13:26:34.442 INFO:teuthology.orchestra.run.smithi029.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2024-01-24T13:26:34.442 INFO:teuthology.orchestra.run.smithi029.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-01-24T13:26:34.442 INFO:teuthology.orchestra.run.smithi029.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2024-01-24T13:26:34.442 INFO:teuthology.orchestra.run.smithi029.stdout:Enabling module streams: 2024-01-24T13:26:34.442 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql 10 2024-01-24T13:26:34.442 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:26:34.442 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-01-24T13:26:34.442 INFO:teuthology.orchestra.run.smithi029.stdout:=========================================================================================================================== 2024-01-24T13:26:34.442 INFO:teuthology.orchestra.run.smithi029.stdout:Install 129 Packages 2024-01-24T13:26:34.442 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrade 16 Packages 2024-01-24T13:26:34.443 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:26:34.443 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 212 M 2024-01-24T13:26:34.443 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-01-24T13:26:34.590 INFO:teuthology.orchestra.run.smithi187.stdout:(16/145): libradosstriper1-18.2.1-341.g4e3182a6 1.9 MB/s | 474 kB 00:00 2024-01-24T13:26:34.811 INFO:teuthology.orchestra.run.smithi029.stdout:(1/145): ceph-18.2.1-341.g4e3182a6.el8.x86_64.r 27 kB/s | 6.4 kB 00:00 2024-01-24T13:26:35.524 INFO:teuthology.orchestra.run.smithi187.stdout:(17/145): ceph-osd-18.2.1-341.g4e3182a6.el8.x86 4.3 MB/s | 18 MB 00:04 2024-01-24T13:26:35.632 INFO:teuthology.orchestra.run.smithi187.stdout:(18/145): python3-ceph-argparse-18.2.1-341.g4e3 413 kB/s | 45 kB 00:00 2024-01-24T13:26:35.678 INFO:teuthology.orchestra.run.smithi029.stdout:(2/145): ceph-base-18.2.1-341.g4e3182a6.el8.x86 4.9 MB/s | 5.4 MB 00:01 2024-01-24T13:26:35.716 INFO:teuthology.orchestra.run.smithi187.stdout:(19/145): python3-ceph-common-18.2.1-341.g4e318 1.4 MB/s | 123 kB 00:00 2024-01-24T13:26:35.717 INFO:teuthology.orchestra.run.smithi163.stdout:(7/145): ceph-mon-18.2.1-341.g4e3182a6.el8.x86_ 2.5 MB/s | 4.6 MB 00:01 2024-01-24T13:26:35.737 INFO:teuthology.orchestra.run.smithi029.stdout:(3/145): ceph-immutable-object-cache-18.2.1-341 2.8 MB/s | 165 kB 00:00 2024-01-24T13:26:35.779 INFO:teuthology.orchestra.run.smithi029.stdout:(4/145): ceph-fuse-18.2.1-341.g4e3182a6.el8.x86 881 kB/s | 852 kB 00:00 2024-01-24T13:26:35.799 INFO:teuthology.orchestra.run.smithi187.stdout:(20/145): python3-cephfs-18.2.1-341.g4e3182a6.e 2.3 MB/s | 197 kB 00:00 2024-01-24T13:26:36.073 INFO:teuthology.orchestra.run.smithi029.stdout:(5/145): ceph-mds-18.2.1-341.g4e3182a6.el8.x86_ 6.8 MB/s | 2.3 MB 00:00 2024-01-24T13:26:36.108 INFO:teuthology.orchestra.run.smithi187.stdout:(21/145): python3-rados-18.2.1-341.g4e3182a6.el 1.2 MB/s | 371 kB 00:00 2024-01-24T13:26:36.308 INFO:teuthology.orchestra.run.smithi187.stdout:(22/145): python3-rbd-18.2.1-341.g4e3182a6.el8. 1.8 MB/s | 365 kB 00:00 2024-01-24T13:26:36.417 INFO:teuthology.orchestra.run.smithi187.stdout:(23/145): python3-rgw-18.2.1-341.g4e3182a6.el8. 984 kB/s | 107 kB 00:00 2024-01-24T13:26:36.534 INFO:teuthology.orchestra.run.smithi187.stdout:(24/145): rbd-fuse-18.2.1-341.g4e3182a6.el8.x86 809 kB/s | 94 kB 00:00 2024-01-24T13:26:36.709 INFO:teuthology.orchestra.run.smithi187.stdout:(25/145): librgw2-18.2.1-341.g4e3182a6.el8.x86_ 3.1 MB/s | 6.5 MB 00:02 2024-01-24T13:26:36.818 INFO:teuthology.orchestra.run.smithi187.stdout:(26/145): rbd-nbd-18.2.1-341.g4e3182a6.el8.x86_ 1.6 MB/s | 181 kB 00:00 2024-01-24T13:26:36.866 INFO:teuthology.orchestra.run.smithi029.stdout:(6/145): ceph-mon-18.2.1-341.g4e3182a6.el8.x86_ 5.8 MB/s | 4.6 MB 00:00 2024-01-24T13:26:36.902 INFO:teuthology.orchestra.run.smithi187.stdout:(27/145): ceph-grafana-dashboards-18.2.1-341.g4 277 kB/s | 23 kB 00:00 2024-01-24T13:26:36.907 INFO:teuthology.orchestra.run.smithi029.stdout:(7/145): ceph-mgr-18.2.1-341.g4e3182a6.el8.x86_ 1.4 MB/s | 1.6 MB 00:01 2024-01-24T13:26:36.993 INFO:teuthology.orchestra.run.smithi187.stdout:(28/145): ceph-mgr-cephadm-18.2.1-341.g4e3182a6 1.4 MB/s | 132 kB 00:00 2024-01-24T13:26:37.585 INFO:teuthology.orchestra.run.smithi187.stdout:(29/145): ceph-mgr-dashboard-18.2.1-341.g4e3182 3.0 MB/s | 1.8 MB 00:00 2024-01-24T13:26:37.901 INFO:teuthology.orchestra.run.smithi163.stdout:(8/145): ceph-common-18.2.1-341.g4e3182a6.el8.x 3.4 MB/s | 22 MB 00:06 2024-01-24T13:26:37.992 INFO:teuthology.orchestra.run.smithi163.stdout:(9/145): ceph-selinux-18.2.1-341.g4e3182a6.el8. 266 kB/s | 24 kB 00:00 2024-01-24T13:26:38.228 INFO:teuthology.orchestra.run.smithi187.stdout:(30/145): rbd-mirror-18.2.1-341.g4e3182a6.el8.x 2.0 MB/s | 3.4 MB 00:01 2024-01-24T13:26:38.436 INFO:teuthology.orchestra.run.smithi187.stdout:(31/145): ceph-mgr-modules-core-18.2.1-341.g4e3 1.1 MB/s | 242 kB 00:00 2024-01-24T13:26:38.536 INFO:teuthology.orchestra.run.smithi187.stdout:(32/145): ceph-mgr-rook-18.2.1-341.g4e3182a6.el 503 kB/s | 50 kB 00:00 2024-01-24T13:26:38.645 INFO:teuthology.orchestra.run.smithi187.stdout:(33/145): ceph-prometheus-alerts-18.2.1-341.g4e 134 kB/s | 15 kB 00:00 2024-01-24T13:26:38.845 INFO:teuthology.orchestra.run.smithi187.stdout:(34/145): ceph-volume-18.2.1-341.g4e3182a6.el8. 1.3 MB/s | 262 kB 00:00 2024-01-24T13:26:39.045 INFO:teuthology.orchestra.run.smithi187.stdout:(35/145): cephadm-18.2.1-341.g4e3182a6.el8.noar 854 kB/s | 171 kB 00:00 2024-01-24T13:26:39.162 INFO:teuthology.orchestra.run.smithi187.stdout:(36/145): cephfs-top-18.2.1-341.g4e3182a6.el8.n 154 kB/s | 18 kB 00:00 2024-01-24T13:26:39.288 INFO:teuthology.orchestra.run.smithi187.stdout:(37/145): gperftools-libs-2.7-9.el8.x86_64.rpm 2.4 MB/s | 306 kB 00:00 2024-01-24T13:26:39.330 INFO:teuthology.orchestra.run.smithi187.stdout:(38/145): liboath-2.6.2-3.el8.x86_64.rpm 1.4 MB/s | 59 kB 00:00 2024-01-24T13:26:39.392 INFO:teuthology.orchestra.run.smithi187.stdout:(39/145): libunwind-1.3.1-3.el8.x86_64.rpm 1.2 MB/s | 75 kB 00:00 2024-01-24T13:26:39.455 INFO:teuthology.orchestra.run.smithi187.stdout:(40/145): python3-asyncssh-2.7.0-2.el8.noarch.r 7.4 MB/s | 476 kB 00:00 2024-01-24T13:26:39.497 INFO:teuthology.orchestra.run.smithi187.stdout:(41/145): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 1.0 MB/s | 44 kB 00:00 2024-01-24T13:26:39.922 INFO:teuthology.orchestra.run.smithi187.stdout:(42/145): ceph-test-18.2.1-341.g4e3182a6.el8.x8 6.6 MB/s | 47 MB 00:07 2024-01-24T13:26:39.924 INFO:teuthology.orchestra.run.smithi029.stdout:(8/145): ceph-osd-18.2.1-341.g4e3182a6.el8.x86_ 5.8 MB/s | 18 MB 00:03 2024-01-24T13:26:39.947 INFO:teuthology.orchestra.run.smithi187.stdout:(43/145): python3-beautifulsoup4-4.6.3-2.el8.1. 411 kB/s | 185 kB 00:00 2024-01-24T13:26:39.975 INFO:teuthology.orchestra.run.smithi029.stdout:(9/145): ceph-selinux-18.2.1-341.g4e3182a6.el8. 487 kB/s | 24 kB 00:00 2024-01-24T13:26:40.040 INFO:teuthology.orchestra.run.smithi187.stdout:(44/145): ceph-mgr-diskprediction-local-18.2.1- 3.0 MB/s | 7.4 MB 00:02 2024-01-24T13:26:40.056 INFO:teuthology.orchestra.run.smithi187.stdout:(45/145): python3-certifi-2018.10.15-7.el8.noar 150 kB/s | 16 kB 00:00 2024-01-24T13:26:40.073 INFO:teuthology.orchestra.run.smithi187.stdout:(46/145): python3-cachetools-3.1.1-4.el8.noarch 219 kB/s | 33 kB 00:00 2024-01-24T13:26:40.115 INFO:teuthology.orchestra.run.smithi187.stdout:(47/145): python3-cherrypy-18.4.0-1.el8.noarch. 6.4 MB/s | 384 kB 00:00 2024-01-24T13:26:40.140 INFO:teuthology.orchestra.run.smithi187.stdout:(48/145): python3-google-auth-1.1.1-10.el8.noar 1.2 MB/s | 81 kB 00:00 2024-01-24T13:26:40.157 INFO:teuthology.orchestra.run.smithi187.stdout:(49/145): python3-jaraco-6.2-6.el8.noarch.rpm 266 kB/s | 11 kB 00:00 2024-01-24T13:26:40.183 INFO:teuthology.orchestra.run.smithi187.stdout:(50/145): python3-jaraco-functools-2.0-4.el8.no 420 kB/s | 18 kB 00:00 2024-01-24T13:26:40.208 INFO:teuthology.orchestra.run.smithi187.stdout:(51/145): python3-cheroot-8.5.2-1.el8.noarch.rp 1.0 MB/s | 173 kB 00:00 2024-01-24T13:26:40.233 INFO:teuthology.orchestra.run.smithi187.stdout:(52/145): python3-logutils-0.3.5-11.el8.noarch. 973 kB/s | 49 kB 00:00 2024-01-24T13:26:40.259 INFO:teuthology.orchestra.run.smithi187.stdout:(53/145): python3-more-itertools-7.2.0-3.el8.no 1.1 MB/s | 59 kB 00:00 2024-01-24T13:26:40.351 INFO:teuthology.orchestra.run.smithi163.stdout:(10/145): ceph-osd-18.2.1-341.g4e3182a6.el8.x86 2.9 MB/s | 18 MB 00:06 2024-01-24T13:26:40.418 INFO:teuthology.orchestra.run.smithi163.stdout:(11/145): cephfs-mirror-18.2.1-341.g4e3182a6.el 3.8 MB/s | 259 kB 00:00 2024-01-24T13:26:40.459 INFO:teuthology.orchestra.run.smithi163.stdout:(12/145): libcephfs-devel-18.2.1-341.g4e3182a6. 747 kB/s | 31 kB 00:00 2024-01-24T13:26:40.484 INFO:teuthology.orchestra.run.smithi187.stdout:(54/145): python3-natsort-7.1.1-2.el8.noarch.rp 241 kB/s | 60 kB 00:00 2024-01-24T13:26:40.618 INFO:teuthology.orchestra.run.smithi163.stdout:(13/145): libcephfs2-18.2.1-341.g4e3182a6.el8.x 4.8 MB/s | 776 kB 00:00 2024-01-24T13:26:40.634 INFO:teuthology.orchestra.run.smithi029.stdout:(10/145): ceph-common-18.2.1-341.g4e3182a6.el8. 3.6 MB/s | 22 MB 00:06 2024-01-24T13:26:40.668 INFO:teuthology.orchestra.run.smithi163.stdout:(14/145): libcephsqlite-18.2.1-341.g4e3182a6.el 2.8 MB/s | 141 kB 00:00 2024-01-24T13:26:40.684 INFO:teuthology.orchestra.run.smithi187.stdout:(55/145): python3-pecan-1.3.2-9.el8.noarch.rpm 665 kB/s | 283 kB 00:00 2024-01-24T13:26:40.709 INFO:teuthology.orchestra.run.smithi187.stdout:(56/145): python3-portend-2.6-1.el8.noarch.rpm 69 kB/s | 16 kB 00:00 2024-01-24T13:26:40.727 INFO:teuthology.orchestra.run.smithi163.stdout:(15/145): librados-devel-18.2.1-341.g4e3182a6.e 2.3 MB/s | 139 kB 00:00 2024-01-24T13:26:40.735 INFO:teuthology.orchestra.run.smithi187.stdout:(57/145): python3-repoze-lru-0.7-6.el8.noarch.r 672 kB/s | 34 kB 00:00 2024-01-24T13:26:40.759 INFO:teuthology.orchestra.run.smithi029.stdout:(11/145): cephfs-mirror-18.2.1-341.g4e3182a6.el 2.0 MB/s | 259 kB 00:00 2024-01-24T13:26:40.760 INFO:teuthology.orchestra.run.smithi187.stdout:(58/145): python3-routes-2.4.1-12.el8.noarch.rp 3.8 MB/s | 196 kB 00:00 2024-01-24T13:26:40.786 INFO:teuthology.orchestra.run.smithi187.stdout:(59/145): python3-rsa-4.9-2.el8.noarch.rpm 1.2 MB/s | 64 kB 00:00 2024-01-24T13:26:40.811 INFO:teuthology.orchestra.run.smithi187.stdout:(60/145): python3-simplegeneric-0.8.1-17.el8.no 372 kB/s | 19 kB 00:00 2024-01-24T13:26:40.835 INFO:teuthology.orchestra.run.smithi029.stdout:(12/145): libcephfs-devel-18.2.1-341.g4e3182a6. 414 kB/s | 31 kB 00:00 2024-01-24T13:26:40.836 INFO:teuthology.orchestra.run.smithi187.stdout:(61/145): python3-singledispatch-3.4.0.3-18.el8 477 kB/s | 24 kB 00:00 2024-01-24T13:26:40.861 INFO:teuthology.orchestra.run.smithi187.stdout:(62/145): python3-tempora-1.14.1-5.el8.noarch.r 584 kB/s | 29 kB 00:00 2024-01-24T13:26:40.887 INFO:teuthology.orchestra.run.smithi187.stdout:(63/145): python3-trustme-0.6.0-4.el8.noarch.rp 549 kB/s | 27 kB 00:00 2024-01-24T13:26:40.920 INFO:teuthology.orchestra.run.smithi187.stdout:(64/145): python3-kubernetes-11.0.0-6.el8.noarc 1.5 MB/s | 1.2 MB 00:00 2024-01-24T13:26:40.936 INFO:teuthology.orchestra.run.smithi163.stdout:(16/145): ceph-radosgw-18.2.1-341.g4e3182a6.el8 2.5 MB/s | 13 MB 00:05 2024-01-24T13:26:40.946 INFO:teuthology.orchestra.run.smithi187.stdout:(65/145): python3-waitress-1.4.3-1.el8.noarch.r 3.0 MB/s | 255 kB 00:00 2024-01-24T13:26:40.961 INFO:teuthology.orchestra.run.smithi029.stdout:(13/145): libcephfs2-18.2.1-341.g4e3182a6.el8.x 6.0 MB/s | 776 kB 00:00 2024-01-24T13:26:40.961 INFO:teuthology.orchestra.run.smithi163.stdout:(17/145): libradosstriper1-18.2.1-341.g4e3182a6 2.0 MB/s | 474 kB 00:00 2024-01-24T13:26:40.971 INFO:teuthology.orchestra.run.smithi187.stdout:(66/145): python3-websocket-client-0.56.0-5.el8 1.2 MB/s | 61 kB 00:00 2024-01-24T13:26:40.996 INFO:teuthology.orchestra.run.smithi187.stdout:(67/145): python3-webob-1.8.5-1.el8.1.noarch.rp 2.3 MB/s | 251 kB 00:00 2024-01-24T13:26:41.036 INFO:teuthology.orchestra.run.smithi163.stdout:(18/145): python3-ceph-argparse-18.2.1-341.g4e3 596 kB/s | 45 kB 00:00 2024-01-24T13:26:41.044 INFO:teuthology.orchestra.run.smithi029.stdout:(14/145): libcephsqlite-18.2.1-341.g4e3182a6.el 1.7 MB/s | 141 kB 00:00 2024-01-24T13:26:41.046 INFO:teuthology.orchestra.run.smithi187.stdout:(68/145): python3-webtest-2.0.33-1.el8.noarch.r 850 kB/s | 85 kB 00:00 2024-01-24T13:26:41.063 INFO:teuthology.orchestra.run.smithi187.stdout:(69/145): python3-zc-lockfile-2.0-2.el8.noarch. 249 kB/s | 23 kB 00:00 2024-01-24T13:26:41.087 INFO:teuthology.orchestra.run.smithi163.stdout:(19/145): python3-ceph-common-18.2.1-341.g4e318 2.4 MB/s | 123 kB 00:00 2024-01-24T13:26:41.088 INFO:teuthology.orchestra.run.smithi187.stdout:(70/145): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.6 MB/s | 70 kB 00:00 2024-01-24T13:26:41.119 INFO:teuthology.orchestra.run.smithi029.stdout:(15/145): librados-devel-18.2.1-341.g4e3182a6.e 1.8 MB/s | 139 kB 00:00 2024-01-24T13:26:41.147 INFO:teuthology.orchestra.run.smithi187.stdout:(71/145): thrift-0.13.0-2.el8.x86_64.rpm 12 MB/s | 1.7 MB 00:00 2024-01-24T13:26:41.162 INFO:teuthology.orchestra.run.smithi163.stdout:(20/145): python3-cephfs-18.2.1-341.g4e3182a6.e 2.6 MB/s | 197 kB 00:00 2024-01-24T13:26:41.214 INFO:teuthology.orchestra.run.smithi187.stdout:(72/145): python3-jsonpointer-1.10-11.el8.noarc 133 kB/s | 20 kB 00:00 2024-01-24T13:26:41.231 INFO:teuthology.orchestra.run.smithi187.stdout:(73/145): python3-prettytable-0.7.2-14.el8.noar 309 kB/s | 44 kB 00:00 2024-01-24T13:26:41.253 INFO:teuthology.orchestra.run.smithi029.stdout:(16/145): libradosstriper1-18.2.1-341.g4e3182a6 3.5 MB/s | 474 kB 00:00 2024-01-24T13:26:41.256 INFO:teuthology.orchestra.run.smithi187.stdout:(74/145): python3-pyasn1-modules-0.3.7-6.el8.no 1.0 MB/s | 110 kB 00:00 2024-01-24T13:26:41.288 INFO:teuthology.orchestra.run.smithi163.stdout:(21/145): python3-rados-18.2.1-341.g4e3182a6.el 2.9 MB/s | 371 kB 00:00 2024-01-24T13:26:41.307 INFO:teuthology.orchestra.run.smithi187.stdout:(75/145): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.6 MB/s | 126 kB 00:00 2024-01-24T13:26:41.332 INFO:teuthology.orchestra.run.smithi187.stdout:(76/145): python3-jsonpatch-1.21-2.el8.noarch.r 226 kB/s | 27 kB 00:00 2024-01-24T13:26:41.365 INFO:teuthology.orchestra.run.smithi187.stdout:(77/145): python3-werkzeug-0.12.2-4.el8.noarch. 4.1 MB/s | 457 kB 00:00 2024-01-24T13:26:41.441 INFO:teuthology.orchestra.run.smithi187.stdout:(78/145): bison-3.0.4-10.el8.x86_64.rpm 5.0 MB/s | 688 kB 00:00 2024-01-24T13:26:41.466 INFO:teuthology.orchestra.run.smithi187.stdout:(79/145): flex-2.6.1-9.el8.x86_64.rpm 2.3 MB/s | 320 kB 00:00 2024-01-24T13:26:41.471 INFO:teuthology.orchestra.run.smithi163.stdout:(22/145): python3-rbd-18.2.1-341.g4e3182a6.el8. 1.9 MB/s | 365 kB 00:00 2024-01-24T13:26:41.508 INFO:teuthology.orchestra.run.smithi187.stdout:(80/145): libtool-2.4.6-25.el8.x86_64.rpm 4.9 MB/s | 709 kB 00:00 2024-01-24T13:26:41.546 INFO:teuthology.orchestra.run.smithi163.stdout:(23/145): python3-rgw-18.2.1-341.g4e3182a6.el8. 1.4 MB/s | 107 kB 00:00 2024-01-24T13:26:41.566 INFO:teuthology.orchestra.run.smithi187.stdout:(81/145): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.5 MB/s | 259 kB 00:00 2024-01-24T13:26:41.591 INFO:teuthology.orchestra.run.smithi187.stdout:(82/145): python3-markupsafe-0.23-19.el8.x86_64 257 kB/s | 39 kB 00:00 2024-01-24T13:26:41.613 INFO:teuthology.orchestra.run.smithi163.stdout:(24/145): rbd-fuse-18.2.1-341.g4e3182a6.el8.x86 1.4 MB/s | 94 kB 00:00 2024-01-24T13:26:41.633 INFO:teuthology.orchestra.run.smithi187.stdout:(83/145): oniguruma-6.8.2-2.el8.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-01-24T13:26:41.734 INFO:teuthology.orchestra.run.smithi187.stdout:(84/145): python3-pyOpenSSL-19.0.0-1.el8.noarch 729 kB/s | 103 kB 00:00 2024-01-24T13:26:41.759 INFO:teuthology.orchestra.run.smithi187.stdout:(85/145): uuid-1.6.2-43.el8.x86_64.rpm 333 kB/s | 64 kB 00:00 2024-01-24T13:26:41.884 INFO:teuthology.orchestra.run.smithi187.stdout:(86/145): python3-jinja2-2.10.1-3.el8.noarch.rp 3.5 MB/s | 538 kB 00:00 2024-01-24T13:26:41.993 INFO:teuthology.orchestra.run.smithi187.stdout:(87/145): python3-numpy-1.14.3-10.el8.x86_64.rp 16 MB/s | 3.7 MB 00:00 2024-01-24T13:26:42.065 INFO:teuthology.orchestra.run.smithi187.stdout:(88/145): socat-1.7.4.1-1.el8.x86_64.rpm 1.8 MB/s | 323 kB 00:00 2024-01-24T13:26:42.135 INFO:teuthology.orchestra.run.smithi187.stdout:(89/145): python3-babel-2.5.1-7.el8.noarch.rpm 9.5 MB/s | 4.8 MB 00:00 2024-01-24T13:26:42.161 INFO:teuthology.orchestra.run.smithi187.stdout:(90/145): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.3 MB/s | 225 kB 00:00 2024-01-24T13:26:42.203 INFO:teuthology.orchestra.run.smithi187.stdout:(91/145): librdkafka-0.11.4-3.el8.x86_64.rpm 2.5 MB/s | 354 kB 00:00 2024-01-24T13:26:42.262 INFO:teuthology.orchestra.run.smithi029.stdout:(17/145): librgw2-18.2.1-341.g4e3182a6.el8.x86_ 6.5 MB/s | 6.5 MB 00:01 2024-01-24T13:26:42.278 INFO:teuthology.orchestra.run.smithi187.stdout:(92/145): python3-mako-1.0.6-14.el8.noarch.rpm 1.3 MB/s | 157 kB 00:00 2024-01-24T13:26:42.346 INFO:teuthology.orchestra.run.smithi029.stdout:(18/145): python3-ceph-argparse-18.2.1-341.g4e3 535 kB/s | 45 kB 00:00 2024-01-24T13:26:42.403 INFO:teuthology.orchestra.run.smithi187.stdout:(93/145): openblas-threads-0.3.15-6.el8.x86_64. 19 MB/s | 5.2 MB 00:00 2024-01-24T13:26:42.421 INFO:teuthology.orchestra.run.smithi029.stdout:(19/145): python3-ceph-common-18.2.1-341.g4e318 1.6 MB/s | 123 kB 00:00 2024-01-24T13:26:42.445 INFO:teuthology.orchestra.run.smithi187.stdout:(94/145): automake-1.16.1-8.el8.noarch.rpm 4.2 MB/s | 714 kB 00:00 2024-01-24T13:26:42.505 INFO:teuthology.orchestra.run.smithi163.stdout:(25/145): rbd-mirror-18.2.1-341.g4e3182a6.el8.x 3.9 MB/s | 3.4 MB 00:00 2024-01-24T13:26:42.514 INFO:teuthology.orchestra.run.smithi029.stdout:(20/145): python3-cephfs-18.2.1-341.g4e3182a6.e 2.1 MB/s | 197 kB 00:00 2024-01-24T13:26:42.555 INFO:teuthology.orchestra.run.smithi163.stdout:(26/145): rbd-nbd-18.2.1-341.g4e3182a6.el8.x86_ 3.6 MB/s | 181 kB 00:00 2024-01-24T13:26:42.569 INFO:teuthology.orchestra.run.smithi187.stdout:(95/145): openblas-0.3.15-6.el8.x86_64.rpm 14 MB/s | 5.0 MB 00:00 2024-01-24T13:26:42.604 INFO:teuthology.orchestra.run.smithi187.stdout:(96/145): libpq-13.11-1.el8.x86_64.rpm 988 kB/s | 198 kB 00:00 2024-01-24T13:26:42.607 INFO:teuthology.orchestra.run.smithi029.stdout:(21/145): python3-rados-18.2.1-341.g4e3182a6.el 3.9 MB/s | 371 kB 00:00 2024-01-24T13:26:42.623 INFO:teuthology.orchestra.run.smithi163.stdout:(27/145): ceph-grafana-dashboards-18.2.1-341.g4 345 kB/s | 23 kB 00:00 2024-01-24T13:26:42.629 INFO:teuthology.orchestra.run.smithi187.stdout:(97/145): jq-1.6-7.el8.x86_64.rpm 1.1 MB/s | 202 kB 00:00 2024-01-24T13:26:42.663 INFO:teuthology.orchestra.run.smithi187.stdout:(98/145): python3-pytz-2017.2-11.el8.noarch.rpm 584 kB/s | 54 kB 00:00 2024-01-24T13:26:42.691 INFO:teuthology.orchestra.run.smithi029.stdout:(22/145): python3-rbd-18.2.1-341.g4e3182a6.el8. 4.3 MB/s | 365 kB 00:00 2024-01-24T13:26:42.714 INFO:teuthology.orchestra.run.smithi163.stdout:(28/145): librgw2-18.2.1-341.g4e3182a6.el8.x86_ 3.7 MB/s | 6.5 MB 00:01 2024-01-24T13:26:42.731 INFO:teuthology.orchestra.run.smithi163.stdout:(29/145): ceph-mgr-cephadm-18.2.1-341.g4e3182a6 1.2 MB/s | 132 kB 00:00 2024-01-24T13:26:42.738 INFO:teuthology.orchestra.run.smithi187.stdout:(99/145): postgresql-contrib-10.23-3.module+el8 7.3 MB/s | 811 kB 00:00 2024-01-24T13:26:42.775 INFO:teuthology.orchestra.run.smithi029.stdout:(23/145): python3-rgw-18.2.1-341.g4e3182a6.el8. 1.2 MB/s | 107 kB 00:00 2024-01-24T13:26:42.780 INFO:teuthology.orchestra.run.smithi187.stdout:(100/145): postgresql-10.23-3.module+el8.9.0+20 8.6 MB/s | 1.5 MB 00:00 2024-01-24T13:26:42.859 INFO:teuthology.orchestra.run.smithi029.stdout:(24/145): rbd-fuse-18.2.1-341.g4e3182a6.el8.x86 1.1 MB/s | 94 kB 00:00 2024-01-24T13:26:42.897 INFO:teuthology.orchestra.run.smithi187.stdout:(101/145): postgresql-server-10.23-3.module+el8 22 MB/s | 5.1 MB 00:00 2024-01-24T13:26:42.956 INFO:teuthology.orchestra.run.smithi187.stdout:(102/145): libverto-devel-0.3.0-5.el8.x86_64.rp 102 kB/s | 18 kB 00:00 2024-01-24T13:26:43.014 INFO:teuthology.orchestra.run.smithi187.stdout:(103/145): python3-jwt-1.6.1-2.el8.noarch.rpm 364 kB/s | 43 kB 00:00 2024-01-24T13:26:43.206 INFO:teuthology.orchestra.run.smithi187.stdout:(104/145): python3-scipy-1.0.0-21.module+el8.9. 29 MB/s | 14 MB 00:00 2024-01-24T13:26:43.223 INFO:teuthology.orchestra.run.smithi163.stdout:(30/145): ceph-mgr-dashboard-18.2.1-341.g4e3182 3.5 MB/s | 1.8 MB 00:00 2024-01-24T13:26:43.231 INFO:teuthology.orchestra.run.smithi187.stdout:(105/145): python3-pycparser-2.14-14.el8.noarch 503 kB/s | 109 kB 00:00 2024-01-24T13:26:43.248 INFO:teuthology.orchestra.run.smithi187.stdout:(106/145): python3-oauthlib-2.1.0-1.el8.noarch. 530 kB/s | 155 kB 00:00 2024-01-24T13:26:43.282 INFO:teuthology.orchestra.run.smithi187.stdout:(107/145): python3-requests-oauthlib-1.0.0-1.el 566 kB/s | 43 kB 00:00 2024-01-24T13:26:43.299 INFO:teuthology.orchestra.run.smithi187.stdout:(108/145): libaio-devel-0.3.112-1.el8.x86_64.rp 278 kB/s | 19 kB 00:00 2024-01-24T13:26:43.324 INFO:teuthology.orchestra.run.smithi187.stdout:(109/145): libzstd-devel-1.4.4-1.el8.x86_64.rpm 581 kB/s | 44 kB 00:00 2024-01-24T13:26:43.332 INFO:teuthology.orchestra.run.smithi163.stdout:(31/145): ceph-mgr-modules-core-18.2.1-341.g4e3 2.2 MB/s | 242 kB 00:00 2024-01-24T13:26:43.359 INFO:teuthology.orchestra.run.smithi187.stdout:(110/145): pcre2-devel-10.32-2.el8.x86_64.rpm 7.8 MB/s | 605 kB 00:00 2024-01-24T13:26:43.374 INFO:teuthology.orchestra.run.smithi187.stdout:(111/145): pcre2-utf32-10.32-2.el8.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-01-24T13:26:43.382 INFO:teuthology.orchestra.run.smithi163.stdout:(32/145): ceph-mgr-rook-18.2.1-341.g4e3182a6.el 1.0 MB/s | 50 kB 00:00 2024-01-24T13:26:43.400 INFO:teuthology.orchestra.run.smithi187.stdout:(112/145): pcre2-utf16-10.32-2.el8.x86_64.rpm 2.2 MB/s | 229 kB 00:00 2024-01-24T13:26:43.424 INFO:teuthology.orchestra.run.smithi163.stdout:(33/145): ceph-prometheus-alerts-18.2.1-341.g4e 349 kB/s | 15 kB 00:00 2024-01-24T13:26:43.467 INFO:teuthology.orchestra.run.smithi187.stdout:(113/145): libselinux-devel-2.9-5.el8.x86_64.rp 1.8 MB/s | 200 kB 00:00 2024-01-24T13:26:43.500 INFO:teuthology.orchestra.run.smithi163.stdout:(34/145): ceph-volume-18.2.1-341.g4e3182a6.el8. 3.4 MB/s | 262 kB 00:00 2024-01-24T13:26:43.517 INFO:teuthology.orchestra.run.smithi187.stdout:(114/145): keyutils-libs-devel-1.5.10-9.el8.x86 338 kB/s | 48 kB 00:00 2024-01-24T13:26:43.591 INFO:teuthology.orchestra.run.smithi163.stdout:(35/145): cephadm-18.2.1-341.g4e3182a6.el8.noar 1.8 MB/s | 171 kB 00:00 2024-01-24T13:26:43.609 INFO:teuthology.orchestra.run.smithi187.stdout:(115/145): libsepol-devel-2.9-3.el8.x86_64.rpm 416 kB/s | 87 kB 00:00 2024-01-24T13:26:43.634 INFO:teuthology.orchestra.run.smithi187.stdout:(116/145): zlib-devel-1.2.11-18.el8_5.x86_64.rp 345 kB/s | 58 kB 00:00 2024-01-24T13:26:43.650 INFO:teuthology.orchestra.run.smithi163.stdout:(36/145): cephfs-top-18.2.1-341.g4e3182a6.el8.n 307 kB/s | 18 kB 00:00 2024-01-24T13:26:43.660 INFO:teuthology.orchestra.run.smithi187.stdout:(117/145): libcom_err-devel-1.45.6-4.el8.x86_64 271 kB/s | 39 kB 00:00 2024-01-24T13:26:43.702 INFO:teuthology.orchestra.run.smithi187.stdout:(118/145): fuse-2.9.7-17.el8.x86_64.rpm 902 kB/s | 83 kB 00:00 2024-01-24T13:26:43.768 INFO:teuthology.orchestra.run.smithi187.stdout:(119/145): libquadmath-8.5.0-20.el8.x86_64.rpm 1.6 MB/s | 172 kB 00:00 2024-01-24T13:26:43.804 INFO:teuthology.orchestra.run.smithi187.stdout:(120/145): libgfortran-8.5.0-20.el8.x86_64.rpm 3.7 MB/s | 645 kB 00:00 2024-01-24T13:26:43.829 INFO:teuthology.orchestra.run.smithi187.stdout:(121/145): libuuid-devel-2.32.1-43.el8.x86_64.r 782 kB/s | 100 kB 00:00 2024-01-24T13:26:43.863 INFO:teuthology.orchestra.run.smithi187.stdout:(122/145): python3-cffi-1.11.5-6.el8.x86_64.rpm 2.5 MB/s | 238 kB 00:00 2024-01-24T13:26:43.876 INFO:teuthology.orchestra.run.smithi029.stdout:(25/145): ceph-radosgw-18.2.1-341.g4e3182a6.el8 1.9 MB/s | 13 MB 00:06 2024-01-24T13:26:43.895 INFO:teuthology.orchestra.run.smithi163.stdout:(37/145): gperftools-libs-2.7-9.el8.x86_64.rpm 1.2 MB/s | 306 kB 00:00 2024-01-24T13:26:43.921 INFO:teuthology.orchestra.run.smithi187.stdout:(123/145): python3-cryptography-3.2.1-6.el8.x86 4.7 MB/s | 559 kB 00:00 2024-01-24T13:26:43.951 INFO:teuthology.orchestra.run.smithi163.stdout:(38/145): liboath-2.6.2-3.el8.x86_64.rpm 1.0 MB/s | 59 kB 00:00 2024-01-24T13:26:43.972 INFO:teuthology.orchestra.run.smithi187.stdout:(124/145): elfutils-libelf-devel-0.189-3.el8.x8 569 kB/s | 62 kB 00:00 2024-01-24T13:26:43.988 INFO:teuthology.orchestra.run.smithi187.stdout:(125/145): xfsprogs-devel-5.0.0-12.el8.x86_64.r 456 kB/s | 73 kB 00:00 2024-01-24T13:26:44.001 INFO:teuthology.orchestra.run.smithi163.stdout:(39/145): libunwind-1.3.1-3.el8.x86_64.rpm 1.5 MB/s | 75 kB 00:00 2024-01-24T13:26:44.035 INFO:teuthology.orchestra.run.smithi029.stdout:(26/145): rbd-mirror-18.2.1-341.g4e3182a6.el8.x 2.9 MB/s | 3.4 MB 00:01 2024-01-24T13:26:44.039 INFO:teuthology.orchestra.run.smithi187.stdout:(126/145): librabbitmq-0.9.0-4.el8.x86_64.rpm 401 kB/s | 47 kB 00:00 2024-01-24T13:26:44.071 INFO:teuthology.orchestra.run.smithi029.stdout:(27/145): rbd-nbd-18.2.1-341.g4e3182a6.el8.x86_ 936 kB/s | 181 kB 00:00 2024-01-24T13:26:44.095 INFO:teuthology.orchestra.run.smithi029.stdout:(28/145): ceph-grafana-dashboards-18.2.1-341.g4 393 kB/s | 23 kB 00:00 2024-01-24T13:26:44.114 INFO:teuthology.orchestra.run.smithi187.stdout:(127/145): krb5-devel-1.18.2-26.el8_9.x86_64.rp 3.9 MB/s | 562 kB 00:00 2024-01-24T13:26:44.137 INFO:teuthology.orchestra.run.smithi029.stdout:(29/145): ceph-mgr-cephadm-18.2.1-341.g4e3182a6 1.9 MB/s | 132 kB 00:00 2024-01-24T13:26:44.181 INFO:teuthology.orchestra.run.smithi187.stdout:(128/145): openssl-devel-1.1.1k-12.el8_9.x86_64 12 MB/s | 2.3 MB 00:00 2024-01-24T13:26:44.198 INFO:teuthology.orchestra.run.smithi187.stdout:(129/145): NetworkManager-initscripts-updown-1. 907 kB/s | 144 kB 00:00 2024-01-24T13:26:44.348 INFO:teuthology.orchestra.run.smithi187.stdout:(130/145): elfutils-libelf-0.189-3.el8.x86_64.r 1.5 MB/s | 232 kB 00:00 2024-01-24T13:26:44.396 INFO:teuthology.orchestra.run.smithi029.stdout:(30/145): ceph-mgr-dashboard-18.2.1-341.g4e3182 5.8 MB/s | 1.8 MB 00:00 2024-01-24T13:26:44.452 INFO:teuthology.orchestra.run.smithi163.stdout:(40/145): ceph-mgr-diskprediction-local-18.2.1- 4.3 MB/s | 7.4 MB 00:01 2024-01-24T13:26:44.480 INFO:teuthology.orchestra.run.smithi029.stdout:(31/145): ceph-mgr-modules-core-18.2.1-341.g4e3 2.8 MB/s | 242 kB 00:00 2024-01-24T13:26:44.515 INFO:teuthology.orchestra.run.smithi187.stdout:(131/145): elfutils-libs-0.189-3.el8.x86_64.rpm 1.8 MB/s | 303 kB 00:00 2024-01-24T13:26:44.552 INFO:teuthology.orchestra.run.smithi163.stdout:(41/145): python3-asyncssh-2.7.0-2.el8.noarch.r 865 kB/s | 476 kB 00:00 2024-01-24T13:26:44.555 INFO:teuthology.orchestra.run.smithi029.stdout:(32/145): ceph-mgr-rook-18.2.1-341.g4e3182a6.el 670 kB/s | 50 kB 00:00 2024-01-24T13:26:44.611 INFO:teuthology.orchestra.run.smithi163.stdout:(42/145): python3-beautifulsoup4-4.6.3-2.el8.1. 3.1 MB/s | 185 kB 00:00 2024-01-24T13:26:44.631 INFO:teuthology.orchestra.run.smithi029.stdout:(33/145): ceph-prometheus-alerts-18.2.1-341.g4e 193 kB/s | 15 kB 00:00 2024-01-24T13:26:44.645 INFO:teuthology.orchestra.run.smithi163.stdout:(43/145): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 227 kB/s | 44 kB 00:00 2024-01-24T13:26:44.657 INFO:teuthology.orchestra.run.smithi187.stdout:(132/145): libmount-2.32.1-43.el8.x86_64.rpm 1.6 MB/s | 237 kB 00:00 2024-01-24T13:26:44.670 INFO:teuthology.orchestra.run.smithi163.stdout:(44/145): python3-cachetools-3.1.1-4.el8.noarch 563 kB/s | 33 kB 00:00 2024-01-24T13:26:44.695 INFO:teuthology.orchestra.run.smithi163.stdout:(45/145): python3-certifi-2018.10.15-7.el8.noar 324 kB/s | 16 kB 00:00 2024-01-24T13:26:44.715 INFO:teuthology.orchestra.run.smithi029.stdout:(34/145): ceph-volume-18.2.1-341.g4e3182a6.el8. 3.1 MB/s | 262 kB 00:00 2024-01-24T13:26:44.729 INFO:teuthology.orchestra.run.smithi163.stdout:(46/145): python3-cheroot-8.5.2-1.el8.noarch.rp 2.9 MB/s | 173 kB 00:00 2024-01-24T13:26:44.788 INFO:teuthology.orchestra.run.smithi163.stdout:(47/145): python3-google-auth-1.1.1-10.el8.noar 1.4 MB/s | 81 kB 00:00 2024-01-24T13:26:44.791 INFO:teuthology.orchestra.run.smithi187.stdout:(133/145): libsmartcols-2.32.1-43.el8.x86_64.rp 1.3 MB/s | 179 kB 00:00 2024-01-24T13:26:44.808 INFO:teuthology.orchestra.run.smithi029.stdout:(35/145): cephadm-18.2.1-341.g4e3182a6.el8.noar 1.8 MB/s | 171 kB 00:00 2024-01-24T13:26:44.813 INFO:teuthology.orchestra.run.smithi163.stdout:(48/145): python3-cherrypy-18.4.0-1.el8.noarch. 3.2 MB/s | 384 kB 00:00 2024-01-24T13:26:44.863 INFO:teuthology.orchestra.run.smithi163.stdout:(49/145): python3-jaraco-6.2-6.el8.noarch.rpm 147 kB/s | 11 kB 00:00 2024-01-24T13:26:44.889 INFO:teuthology.orchestra.run.smithi163.stdout:(50/145): python3-jaraco-functools-2.0-4.el8.no 233 kB/s | 18 kB 00:00 2024-01-24T13:26:44.930 INFO:teuthology.orchestra.run.smithi163.stdout:(51/145): python3-logutils-0.3.5-11.el8.noarch. 1.2 MB/s | 49 kB 00:00 2024-01-24T13:26:44.932 INFO:teuthology.orchestra.run.smithi187.stdout:(134/145): libuuid-2.32.1-43.el8.x86_64.rpm 700 kB/s | 99 kB 00:00 2024-01-24T13:26:44.973 INFO:teuthology.orchestra.run.smithi163.stdout:(52/145): python3-kubernetes-11.0.0-6.el8.noarc 11 MB/s | 1.2 MB 00:00 2024-01-24T13:26:44.998 INFO:teuthology.orchestra.run.smithi163.stdout:(53/145): python3-more-itertools-7.2.0-3.el8.no 875 kB/s | 59 kB 00:00 2024-01-24T13:26:45.023 INFO:teuthology.orchestra.run.smithi163.stdout:(54/145): python3-natsort-7.1.1-2.el8.noarch.rp 1.2 MB/s | 60 kB 00:00 2024-01-24T13:26:45.024 INFO:teuthology.orchestra.run.smithi187.stdout:(135/145): librbd1-18.2.1-341.g4e3182a6.el8.x86 4.2 MB/s | 3.6 MB 00:00 2024-01-24T13:26:45.057 INFO:teuthology.orchestra.run.smithi163.stdout:(55/145): python3-pecan-1.3.2-9.el8.noarch.rpm 4.7 MB/s | 283 kB 00:00 2024-01-24T13:26:45.082 INFO:teuthology.orchestra.run.smithi163.stdout:(56/145): python3-portend-2.6-1.el8.noarch.rpm 266 kB/s | 16 kB 00:00 2024-01-24T13:26:45.107 INFO:teuthology.orchestra.run.smithi163.stdout:(57/145): python3-repoze-lru-0.7-6.el8.noarch.r 668 kB/s | 34 kB 00:00 2024-01-24T13:26:45.133 INFO:teuthology.orchestra.run.smithi163.stdout:(58/145): python3-routes-2.4.1-12.el8.noarch.rp 3.8 MB/s | 196 kB 00:00 2024-01-24T13:26:45.150 INFO:teuthology.orchestra.run.smithi187.stdout:(136/145): util-linux-user-2.32.1-43.el8.x86_64 821 kB/s | 103 kB 00:00 2024-01-24T13:26:45.192 INFO:teuthology.orchestra.run.smithi163.stdout:(59/145): python3-rsa-4.9-2.el8.noarch.rpm 767 kB/s | 64 kB 00:00 2024-01-24T13:26:45.200 INFO:teuthology.orchestra.run.smithi187.stdout:(137/145): util-linux-2.32.1-43.el8.x86_64.rpm 9.3 MB/s | 2.5 MB 00:00 2024-01-24T13:26:45.217 INFO:teuthology.orchestra.run.smithi163.stdout:(60/145): python3-simplegeneric-0.8.1-17.el8.no 224 kB/s | 19 kB 00:00 2024-01-24T13:26:45.242 INFO:teuthology.orchestra.run.smithi163.stdout:(61/145): python3-singledispatch-3.4.0.3-18.el8 477 kB/s | 24 kB 00:00 2024-01-24T13:26:45.267 INFO:teuthology.orchestra.run.smithi163.stdout:(62/145): python3-tempora-1.14.1-5.el8.noarch.r 582 kB/s | 29 kB 00:00 2024-01-24T13:26:45.292 INFO:teuthology.orchestra.run.smithi187.stdout:(138/145): iproute-6.2.0-5.el8_9.x86_64.rpm 5.9 MB/s | 854 kB 00:00 2024-01-24T13:26:45.293 INFO:teuthology.orchestra.run.smithi163.stdout:(63/145): python3-trustme-0.6.0-4.el8.noarch.rp 547 kB/s | 27 kB 00:00 2024-01-24T13:26:45.318 INFO:teuthology.orchestra.run.smithi163.stdout:(64/145): python3-waitress-1.4.3-1.el8.noarch.r 5.0 MB/s | 255 kB 00:00 2024-01-24T13:26:45.342 INFO:teuthology.orchestra.run.smithi187.stdout:(139/145): librados2-18.2.1-341.g4e3182a6.el8.x 3.0 MB/s | 3.7 MB 00:01 2024-01-24T13:26:45.354 INFO:teuthology.orchestra.run.smithi163.stdout:(65/145): python3-webob-1.8.5-1.el8.1.noarch.rp 4.0 MB/s | 251 kB 00:00 2024-01-24T13:26:45.368 INFO:teuthology.orchestra.run.smithi187.stdout:(140/145): libblkid-2.32.1-43.el8.x86_64.rpm 1.3 MB/s | 220 kB 00:00 2024-01-24T13:26:45.379 INFO:teuthology.orchestra.run.smithi163.stdout:(66/145): python3-websocket-client-0.56.0-5.el8 1.0 MB/s | 61 kB 00:00 2024-01-24T13:26:45.401 INFO:teuthology.orchestra.run.smithi187.stdout:(141/145): libfdisk-2.32.1-43.el8.x86_64.rpm 2.3 MB/s | 254 kB 00:00 2024-01-24T13:26:45.404 INFO:teuthology.orchestra.run.smithi163.stdout:(67/145): python3-webtest-2.0.33-1.el8.noarch.r 1.7 MB/s | 85 kB 00:00 2024-01-24T13:26:45.430 INFO:teuthology.orchestra.run.smithi163.stdout:(68/145): python3-zc-lockfile-2.0-2.el8.noarch. 459 kB/s | 23 kB 00:00 2024-01-24T13:26:45.480 INFO:teuthology.orchestra.run.smithi163.stdout:(69/145): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 70 kB 00:00 2024-01-24T13:26:45.485 INFO:teuthology.orchestra.run.smithi187.stdout:(142/145): openssl-1.1.1k-12.el8_9.x86_64.rpm 5.9 MB/s | 711 kB 00:00 2024-01-24T13:26:45.526 INFO:teuthology.orchestra.run.smithi187.stdout:(143/145): xfsprogs-5.0.0-12.el8.x86_64.rpm 5.8 MB/s | 1.1 MB 00:00 2024-01-24T13:26:45.555 INFO:teuthology.orchestra.run.smithi163.stdout:(70/145): thrift-0.13.0-2.el8.x86_64.rpm 12 MB/s | 1.7 MB 00:00 2024-01-24T13:26:45.600 INFO:teuthology.orchestra.run.smithi187.stdout:(144/145): openssl-libs-1.1.1k-12.el8_9.x86_64. 7.4 MB/s | 1.5 MB 00:00 2024-01-24T13:26:45.635 INFO:teuthology.orchestra.run.smithi187.stdout:(145/145): xfsdump-3.1.8-7.el8_9.x86_64.rpm 2.2 MB/s | 333 kB 00:00 2024-01-24T13:26:45.638 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-01-24T13:26:45.638 INFO:teuthology.orchestra.run.smithi187.stdout:Total 14 MB/s | 212 MB 00:15 2024-01-24T13:26:45.651 INFO:teuthology.orchestra.run.smithi029.stdout:(36/145): ceph-test-18.2.1-341.g4e3182a6.el8.x8 8.2 MB/s | 47 MB 00:05 2024-01-24T13:26:45.676 INFO:teuthology.orchestra.run.smithi029.stdout:(37/145): cephfs-top-18.2.1-341.g4e3182a6.el8.n 21 kB/s | 18 kB 00:00 2024-01-24T13:26:45.680 INFO:teuthology.orchestra.run.smithi163.stdout:(71/145): python3-jsonpointer-1.10-11.el8.noarc 100 kB/s | 20 kB 00:00 2024-01-24T13:26:45.731 INFO:teuthology.orchestra.run.smithi163.stdout:(72/145): python3-prettytable-0.7.2-14.el8.noar 251 kB/s | 44 kB 00:00 2024-01-24T13:26:45.815 INFO:teuthology.orchestra.run.smithi163.stdout:(73/145): python3-pyasn1-modules-0.3.7-6.el8.no 824 kB/s | 110 kB 00:00 2024-01-24T13:26:45.840 INFO:teuthology.orchestra.run.smithi163.stdout:(74/145): python3-jsonpatch-1.21-2.el8.noarch.r 245 kB/s | 27 kB 00:00 2024-01-24T13:26:45.868 INFO:teuthology.orchestra.run.smithi029.stdout:(38/145): liboath-2.6.2-3.el8.x86_64.rpm 307 kB/s | 59 kB 00:00 2024-01-24T13:26:45.902 INFO:teuthology.orchestra.run.smithi029.stdout:(39/145): gperftools-libs-2.7-9.el8.x86_64.rpm 1.2 MB/s | 306 kB 00:00 2024-01-24T13:26:45.928 INFO:teuthology.orchestra.run.smithi029.stdout:(40/145): libunwind-1.3.1-3.el8.x86_64.rpm 1.2 MB/s | 75 kB 00:00 2024-01-24T13:26:45.948 INFO:teuthology.orchestra.run.smithi163.stdout:(75/145): python3-pyasn1-0.3.7-6.el8.noarch.rpm 946 kB/s | 126 kB 00:00 2024-01-24T13:26:45.970 INFO:teuthology.orchestra.run.smithi029.stdout:(41/145): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 1.0 MB/s | 44 kB 00:00 2024-01-24T13:26:45.999 INFO:teuthology.orchestra.run.smithi163.stdout:(76/145): python3-werkzeug-0.12.2-4.el8.noarch. 2.8 MB/s | 457 kB 00:00 2024-01-24T13:26:46.004 INFO:teuthology.orchestra.run.smithi029.stdout:(42/145): python3-asyncssh-2.7.0-2.el8.noarch.r 4.6 MB/s | 476 kB 00:00 2024-01-24T13:26:46.038 INFO:teuthology.orchestra.run.smithi029.stdout:(43/145): python3-beautifulsoup4-4.6.3-2.el8.1. 2.7 MB/s | 185 kB 00:00 2024-01-24T13:26:46.057 INFO:teuthology.orchestra.run.smithi163.stdout:(77/145): bison-3.0.4-10.el8.x86_64.rpm 6.2 MB/s | 688 kB 00:00 2024-01-24T13:26:46.063 INFO:teuthology.orchestra.run.smithi029.stdout:(44/145): python3-cachetools-3.1.1-4.el8.noarch 561 kB/s | 33 kB 00:00 2024-01-24T13:26:46.097 INFO:teuthology.orchestra.run.smithi029.stdout:(45/145): python3-certifi-2018.10.15-7.el8.noar 275 kB/s | 16 kB 00:00 2024-01-24T13:26:46.158 INFO:teuthology.orchestra.run.smithi163.stdout:(78/145): flex-2.6.1-9.el8.x86_64.rpm 2.0 MB/s | 320 kB 00:00 2024-01-24T13:26:46.173 INFO:teuthology.orchestra.run.smithi029.stdout:(46/145): python3-cheroot-8.5.2-1.el8.noarch.rp 1.5 MB/s | 173 kB 00:00 2024-01-24T13:26:46.191 INFO:teuthology.orchestra.run.smithi163.stdout:(79/145): libtool-2.4.6-25.el8.x86_64.rpm 5.2 MB/s | 709 kB 00:00 2024-01-24T13:26:46.206 INFO:teuthology.orchestra.run.smithi029.stdout:(47/145): python3-cherrypy-18.4.0-1.el8.noarch. 3.4 MB/s | 384 kB 00:00 2024-01-24T13:26:46.232 INFO:teuthology.orchestra.run.smithi029.stdout:(48/145): python3-google-auth-1.1.1-10.el8.noar 1.3 MB/s | 81 kB 00:00 2024-01-24T13:26:46.258 INFO:teuthology.orchestra.run.smithi029.stdout:(49/145): python3-jaraco-6.2-6.el8.noarch.rpm 218 kB/s | 11 kB 00:00 2024-01-24T13:26:46.283 INFO:teuthology.orchestra.run.smithi163.stdout:(80/145): python3-markupsafe-0.23-19.el8.x86_64 310 kB/s | 39 kB 00:00 2024-01-24T13:26:46.285 INFO:teuthology.orchestra.run.smithi029.stdout:(50/145): python3-jaraco-functools-2.0-4.el8.no 337 kB/s | 18 kB 00:00 2024-01-24T13:26:46.308 INFO:teuthology.orchestra.run.smithi163.stdout:(81/145): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.2 MB/s | 259 kB 00:00 2024-01-24T13:26:46.344 INFO:teuthology.orchestra.run.smithi029.stdout:(51/145): python3-logutils-0.3.5-11.el8.noarch. 831 kB/s | 49 kB 00:00 2024-01-24T13:26:46.394 INFO:teuthology.orchestra.run.smithi029.stdout:(52/145): python3-kubernetes-11.0.0-6.el8.noarc 8.6 MB/s | 1.2 MB 00:00 2024-01-24T13:26:46.428 INFO:teuthology.orchestra.run.smithi029.stdout:(53/145): python3-more-itertools-7.2.0-3.el8.no 698 kB/s | 59 kB 00:00 2024-01-24T13:26:46.454 INFO:teuthology.orchestra.run.smithi029.stdout:(54/145): python3-natsort-7.1.1-2.el8.noarch.rp 1.0 MB/s | 60 kB 00:00 2024-01-24T13:26:46.479 INFO:teuthology.orchestra.run.smithi029.stdout:(55/145): python3-pecan-1.3.2-9.el8.noarch.rpm 5.5 MB/s | 283 kB 00:00 2024-01-24T13:26:46.492 INFO:teuthology.orchestra.run.smithi163.stdout:(82/145): oniguruma-6.8.2-2.el8.x86_64.rpm 899 kB/s | 187 kB 00:00 2024-01-24T13:26:46.542 INFO:teuthology.orchestra.run.smithi163.stdout:(83/145): uuid-1.6.2-43.el8.x86_64.rpm 274 kB/s | 64 kB 00:00 2024-01-24T13:26:46.626 INFO:teuthology.orchestra.run.smithi163.stdout:(84/145): python3-pyOpenSSL-19.0.0-1.el8.noarch 774 kB/s | 103 kB 00:00 2024-01-24T13:26:46.663 INFO:teuthology.orchestra.run.smithi029.stdout:(56/145): ceph-mgr-diskprediction-local-18.2.1- 2.9 MB/s | 7.4 MB 00:02 2024-01-24T13:26:46.678 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-01-24T13:26:46.697 INFO:teuthology.orchestra.run.smithi029.stdout:(57/145): python3-portend-2.6-1.el8.noarch.rpm 64 kB/s | 16 kB 00:00 2024-01-24T13:26:46.723 INFO:teuthology.orchestra.run.smithi029.stdout:(58/145): python3-repoze-lru-0.7-6.el8.noarch.r 138 kB/s | 34 kB 00:00 2024-01-24T13:26:46.748 INFO:teuthology.orchestra.run.smithi029.stdout:(59/145): python3-routes-2.4.1-12.el8.noarch.rp 2.3 MB/s | 196 kB 00:00 2024-01-24T13:26:46.759 INFO:teuthology.orchestra.run.smithi163.stdout:(85/145): python3-babel-2.5.1-7.el8.noarch.rpm 22 MB/s | 4.8 MB 00:00 2024-01-24T13:26:46.774 INFO:teuthology.orchestra.run.smithi029.stdout:(60/145): python3-simplegeneric-0.8.1-17.el8.no 410 kB/s | 19 kB 00:00 2024-01-24T13:26:46.793 INFO:teuthology.orchestra.run.smithi163.stdout:(86/145): python3-jinja2-2.10.1-3.el8.noarch.rp 3.1 MB/s | 538 kB 00:00 2024-01-24T13:26:46.799 INFO:teuthology.orchestra.run.smithi029.stdout:(61/145): python3-singledispatch-3.4.0.3-18.el8 479 kB/s | 24 kB 00:00 2024-01-24T13:26:46.824 INFO:teuthology.orchestra.run.smithi029.stdout:(62/145): python3-tempora-1.14.1-5.el8.noarch.r 581 kB/s | 29 kB 00:00 2024-01-24T13:26:46.858 INFO:teuthology.orchestra.run.smithi029.stdout:(63/145): python3-trustme-0.6.0-4.el8.noarch.rp 468 kB/s | 27 kB 00:00 2024-01-24T13:26:46.884 INFO:teuthology.orchestra.run.smithi029.stdout:(64/145): python3-rsa-4.9-2.el8.noarch.rpm 344 kB/s | 64 kB 00:00 2024-01-24T13:26:46.910 INFO:teuthology.orchestra.run.smithi029.stdout:(65/145): python3-waitress-1.4.3-1.el8.noarch.r 2.9 MB/s | 255 kB 00:00 2024-01-24T13:26:46.936 INFO:teuthology.orchestra.run.smithi029.stdout:(66/145): python3-websocket-client-0.56.0-5.el8 1.1 MB/s | 61 kB 00:00 2024-01-24T13:26:46.971 INFO:teuthology.orchestra.run.smithi029.stdout:(67/145): python3-webtest-2.0.33-1.el8.noarch.r 1.4 MB/s | 85 kB 00:00 2024-01-24T13:26:47.004 INFO:teuthology.orchestra.run.smithi029.stdout:(68/145): python3-webob-1.8.5-1.el8.1.noarch.rp 1.7 MB/s | 251 kB 00:00 2024-01-24T13:26:47.030 INFO:teuthology.orchestra.run.smithi029.stdout:(69/145): python3-zc-lockfile-2.0-2.el8.noarch. 247 kB/s | 23 kB 00:00 2024-01-24T13:26:47.071 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-01-24T13:26:47.071 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-01-24T13:26:47.080 INFO:teuthology.orchestra.run.smithi029.stdout:(70/145): xmlstarlet-1.6.1-20.el8.x86_64.rpm 919 kB/s | 70 kB 00:00 2024-01-24T13:26:47.156 INFO:teuthology.orchestra.run.smithi029.stdout:(71/145): thrift-0.13.0-2.el8.x86_64.rpm 9.4 MB/s | 1.7 MB 00:00 2024-01-24T13:26:47.198 INFO:teuthology.orchestra.run.smithi029.stdout:(72/145): python3-jsonpointer-1.10-11.el8.noarc 119 kB/s | 20 kB 00:00 2024-01-24T13:26:47.260 INFO:teuthology.orchestra.run.smithi163.stdout:(87/145): ceph-test-18.2.1-341.g4e3182a6.el8.x8 5.0 MB/s | 47 MB 00:09 2024-01-24T13:26:47.265 INFO:teuthology.orchestra.run.smithi029.stdout:(73/145): python3-prettytable-0.7.2-14.el8.noar 238 kB/s | 44 kB 00:00 2024-01-24T13:26:47.293 INFO:teuthology.orchestra.run.smithi163.stdout:(88/145): socat-1.7.4.1-1.el8.x86_64.rpm 646 kB/s | 323 kB 00:00 2024-01-24T13:26:47.324 INFO:teuthology.orchestra.run.smithi029.stdout:(74/145): python3-jsonpatch-1.21-2.el8.noarch.r 227 kB/s | 27 kB 00:00 2024-01-24T13:26:47.383 INFO:teuthology.orchestra.run.smithi029.stdout:(75/145): python3-pyasn1-modules-0.3.7-6.el8.no 505 kB/s | 110 kB 00:00 2024-01-24T13:26:47.408 INFO:teuthology.orchestra.run.smithi029.stdout:(76/145): python3-pyasn1-0.3.7-6.el8.noarch.rpm 885 kB/s | 126 kB 00:00 2024-01-24T13:26:47.410 INFO:teuthology.orchestra.run.smithi163.stdout:(89/145): python3-numpy-1.14.3-10.el8.x86_64.rp 5.7 MB/s | 3.7 MB 00:00 2024-01-24T13:26:47.436 INFO:teuthology.orchestra.run.smithi163.stdout:(90/145): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.3 MB/s | 225 kB 00:00 2024-01-24T13:26:47.451 INFO:teuthology.orchestra.run.smithi029.stdout:(77/145): python3-werkzeug-0.12.2-4.el8.noarch. 3.5 MB/s | 457 kB 00:00 2024-01-24T13:26:47.461 INFO:teuthology.orchestra.run.smithi163.stdout:(91/145): librdkafka-0.11.4-3.el8.x86_64.rpm 2.1 MB/s | 354 kB 00:00 2024-01-24T13:26:47.535 INFO:teuthology.orchestra.run.smithi029.stdout:(78/145): bison-3.0.4-10.el8.x86_64.rpm 4.5 MB/s | 688 kB 00:00 2024-01-24T13:26:47.585 INFO:teuthology.orchestra.run.smithi029.stdout:(79/145): libtool-2.4.6-25.el8.x86_64.rpm 5.2 MB/s | 709 kB 00:00 2024-01-24T13:26:47.594 INFO:teuthology.orchestra.run.smithi163.stdout:(92/145): python3-mako-1.0.6-14.el8.noarch.rpm 991 kB/s | 157 kB 00:00 2024-01-24T13:26:47.619 INFO:teuthology.orchestra.run.smithi029.stdout:(80/145): flex-2.6.1-9.el8.x86_64.rpm 1.5 MB/s | 320 kB 00:00 2024-01-24T13:26:47.661 INFO:teuthology.orchestra.run.smithi163.stdout:(93/145): openblas-threads-0.3.15-6.el8.x86_64. 21 MB/s | 5.2 MB 00:00 2024-01-24T13:26:47.711 INFO:teuthology.orchestra.run.smithi029.stdout:(81/145): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.0 MB/s | 259 kB 00:00 2024-01-24T13:26:47.738 INFO:teuthology.orchestra.run.smithi029.stdout:(82/145): python3-markupsafe-0.23-19.el8.x86_64 191 kB/s | 39 kB 00:00 2024-01-24T13:26:47.795 INFO:teuthology.orchestra.run.smithi163.stdout:(94/145): openblas-0.3.15-6.el8.x86_64.rpm 15 MB/s | 5.0 MB 00:00 2024-01-24T13:26:47.813 INFO:teuthology.orchestra.run.smithi029.stdout:(83/145): oniguruma-6.8.2-2.el8.x86_64.rpm 965 kB/s | 187 kB 00:00 2024-01-24T13:26:47.839 INFO:teuthology.orchestra.run.smithi029.stdout:(84/145): uuid-1.6.2-43.el8.x86_64.rpm 500 kB/s | 64 kB 00:00 2024-01-24T13:26:47.854 INFO:teuthology.orchestra.run.smithi163.stdout:(95/145): libpq-13.11-1.el8.x86_64.rpm 1.0 MB/s | 198 kB 00:00 2024-01-24T13:26:47.865 INFO:teuthology.orchestra.run.smithi029.stdout:(85/145): python3-pyOpenSSL-19.0.0-1.el8.noarch 819 kB/s | 103 kB 00:00 2024-01-24T13:26:47.887 INFO:teuthology.orchestra.run.smithi163.stdout:(96/145): automake-1.16.1-8.el8.noarch.rpm 2.4 MB/s | 714 kB 00:00 2024-01-24T13:26:47.938 INFO:teuthology.orchestra.run.smithi163.stdout:(97/145): jq-1.6-7.el8.x86_64.rpm 1.4 MB/s | 202 kB 00:00 2024-01-24T13:26:47.948 INFO:teuthology.orchestra.run.smithi029.stdout:(86/145): python3-jinja2-2.10.1-3.el8.noarch.rp 4.8 MB/s | 538 kB 00:00 2024-01-24T13:26:47.963 INFO:teuthology.orchestra.run.smithi163.stdout:(98/145): python3-pytz-2017.2-11.el8.noarch.rpm 499 kB/s | 54 kB 00:00 2024-01-24T13:26:48.088 INFO:teuthology.orchestra.run.smithi163.stdout:(99/145): postgresql-contrib-10.23-3.module+el8 5.3 MB/s | 811 kB 00:00 2024-01-24T13:26:48.153 INFO:teuthology.orchestra.run.smithi029.stdout:(87/145): python3-babel-2.5.1-7.el8.noarch.rpm 14 MB/s | 4.8 MB 00:00 2024-01-24T13:26:48.214 INFO:teuthology.orchestra.run.smithi163.stdout:(100/145): postgresql-server-10.23-3.module+el8 20 MB/s | 5.1 MB 00:00 2024-01-24T13:26:48.265 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-01-24T13:26:48.272 INFO:teuthology.orchestra.run.smithi163.stdout:(101/145): postgresql-10.23-3.module+el8.9.0+20 3.9 MB/s | 1.5 MB 00:00 2024-01-24T13:26:48.274 INFO:teuthology.orchestra.run.smithi029.stdout:(88/145): python3-numpy-1.14.3-10.el8.x86_64.rp 9.1 MB/s | 3.7 MB 00:00 2024-01-24T13:26:48.275 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-01-24T13:26:48.309 INFO:teuthology.orchestra.run.smithi029.stdout:(89/145): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.4 MB/s | 225 kB 00:00 2024-01-24T13:26:48.342 INFO:teuthology.orchestra.run.smithi029.stdout:(90/145): socat-1.7.4.1-1.el8.x86_64.rpm 821 kB/s | 323 kB 00:00 2024-01-24T13:26:48.364 INFO:teuthology.orchestra.run.smithi163.stdout:(102/145): python3-jwt-1.6.1-2.el8.noarch.rpm 465 kB/s | 43 kB 00:00 2024-01-24T13:26:48.406 INFO:teuthology.orchestra.run.smithi163.stdout:(103/145): libverto-devel-0.3.0-5.el8.x86_64.rp 93 kB/s | 18 kB 00:00 2024-01-24T13:26:48.451 INFO:teuthology.orchestra.run.smithi029.stdout:(91/145): librdkafka-0.11.4-3.el8.x86_64.rpm 2.0 MB/s | 354 kB 00:00 2024-01-24T13:26:48.485 INFO:teuthology.orchestra.run.smithi029.stdout:(92/145): python3-mako-1.0.6-14.el8.noarch.rpm 1.1 MB/s | 157 kB 00:00 2024-01-24T13:26:48.565 INFO:teuthology.orchestra.run.smithi163.stdout:(104/145): python3-scipy-1.0.0-21.module+el8.9. 29 MB/s | 14 MB 00:00 2024-01-24T13:26:48.590 INFO:teuthology.orchestra.run.smithi163.stdout:(105/145): python3-pycparser-2.14-14.el8.noarch 594 kB/s | 109 kB 00:00 2024-01-24T13:26:48.615 INFO:teuthology.orchestra.run.smithi163.stdout:(106/145): python3-oauthlib-2.1.0-1.el8.noarch. 618 kB/s | 155 kB 00:00 2024-01-24T13:26:48.652 INFO:teuthology.orchestra.run.smithi029.stdout:(93/145): openblas-threads-0.3.15-6.el8.x86_64. 15 MB/s | 5.2 MB 00:00 2024-01-24T13:26:48.673 INFO:teuthology.orchestra.run.smithi163.stdout:(107/145): python3-requests-oauthlib-1.0.0-1.el 393 kB/s | 43 kB 00:00 2024-01-24T13:26:48.711 INFO:teuthology.orchestra.run.smithi029.stdout:(94/145): automake-1.16.1-8.el8.noarch.rpm 3.1 MB/s | 714 kB 00:00 2024-01-24T13:26:48.774 INFO:teuthology.orchestra.run.smithi163.stdout:(108/145): libzstd-devel-1.4.4-1.el8.x86_64.rpm 277 kB/s | 44 kB 00:00 2024-01-24T13:26:48.778 INFO:teuthology.orchestra.run.smithi029.stdout:(95/145): libpq-13.11-1.el8.x86_64.rpm 1.5 MB/s | 198 kB 00:00 2024-01-24T13:26:48.799 INFO:teuthology.orchestra.run.smithi163.stdout:(109/145): libaio-devel-0.3.112-1.el8.x86_64.rp 89 kB/s | 19 kB 00:00 2024-01-24T13:26:48.812 INFO:teuthology.orchestra.run.smithi029.stdout:(96/145): jq-1.6-7.el8.x86_64.rpm 2.0 MB/s | 202 kB 00:00 2024-01-24T13:26:48.833 INFO:teuthology.orchestra.run.smithi163.stdout:(110/145): pcre2-devel-10.32-2.el8.x86_64.rpm 3.7 MB/s | 605 kB 00:00 2024-01-24T13:26:48.916 INFO:teuthology.orchestra.run.smithi163.stdout:(111/145): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.8 MB/s | 220 kB 00:00 2024-01-24T13:26:48.931 INFO:teuthology.orchestra.run.smithi029.stdout:(97/145): openblas-0.3.15-6.el8.x86_64.rpm 11 MB/s | 5.0 MB 00:00 2024-01-24T13:26:48.941 INFO:teuthology.orchestra.run.smithi163.stdout:(112/145): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.3 MB/s | 229 kB 00:00 2024-01-24T13:26:48.956 INFO:teuthology.orchestra.run.smithi029.stdout:(98/145): python3-pytz-2017.2-11.el8.noarch.rpm 305 kB/s | 54 kB 00:00 2024-01-24T13:26:48.967 INFO:teuthology.orchestra.run.smithi163.stdout:(113/145): libselinux-devel-2.9-5.el8.x86_64.rp 1.5 MB/s | 200 kB 00:00 2024-01-24T13:26:49.015 INFO:teuthology.orchestra.run.smithi029.stdout:(99/145): postgresql-10.23-3.module+el8.9.0+208 7.4 MB/s | 1.5 MB 00:00 2024-01-24T13:26:49.067 INFO:teuthology.orchestra.run.smithi163.stdout:(114/145): keyutils-libs-devel-1.5.10-9.el8.x86 319 kB/s | 48 kB 00:00 2024-01-24T13:26:49.092 INFO:teuthology.orchestra.run.smithi163.stdout:(115/145): libsepol-devel-2.9-3.el8.x86_64.rpm 578 kB/s | 87 kB 00:00 2024-01-24T13:26:49.109 INFO:teuthology.orchestra.run.smithi163.stdout:(116/145): zlib-devel-1.2.11-18.el8_5.x86_64.rp 405 kB/s | 58 kB 00:00 2024-01-24T13:26:49.149 INFO:teuthology.orchestra.run.smithi029.stdout:(100/145): postgresql-contrib-10.23-3.module+el 3.6 MB/s | 811 kB 00:00 2024-01-24T13:26:49.201 INFO:teuthology.orchestra.run.smithi163.stdout:(117/145): libcom_err-devel-1.45.6-4.el8.x86_64 288 kB/s | 39 kB 00:00 2024-01-24T13:26:49.260 INFO:teuthology.orchestra.run.smithi163.stdout:(118/145): libgfortran-8.5.0-20.el8.x86_64.rpm 4.2 MB/s | 645 kB 00:00 2024-01-24T13:26:49.293 INFO:teuthology.orchestra.run.smithi163.stdout:(119/145): fuse-2.9.7-17.el8.x86_64.rpm 414 kB/s | 83 kB 00:00 2024-01-24T13:26:49.310 INFO:teuthology.orchestra.run.smithi163.stdout:(120/145): libquadmath-8.5.0-20.el8.x86_64.rpm 1.5 MB/s | 172 kB 00:00 2024-01-24T13:26:49.366 INFO:teuthology.orchestra.run.smithi029.stdout:(101/145): postgresql-server-10.23-3.module+el8 12 MB/s | 5.1 MB 00:00 2024-01-24T13:26:49.392 INFO:teuthology.orchestra.run.smithi029.stdout:(102/145): libverto-devel-0.3.0-5.el8.x86_64.rp 74 kB/s | 18 kB 00:00 2024-01-24T13:26:49.394 INFO:teuthology.orchestra.run.smithi163.stdout:(121/145): python3-cffi-1.11.5-6.el8.x86_64.rpm 2.3 MB/s | 238 kB 00:00 2024-01-24T13:26:49.412 INFO:teuthology.orchestra.run.smithi163.stdout:(122/145): libuuid-devel-2.32.1-43.el8.x86_64.r 662 kB/s | 100 kB 00:00 2024-01-24T13:26:49.461 INFO:teuthology.orchestra.run.smithi163.stdout:(123/145): python3-cryptography-3.2.1-6.el8.x86 3.6 MB/s | 559 kB 00:00 2024-01-24T13:26:49.484 INFO:teuthology.orchestra.run.smithi029.stdout:(103/145): python3-jwt-1.6.1-2.el8.noarch.rpm 363 kB/s | 43 kB 00:00 2024-01-24T13:26:49.536 INFO:teuthology.orchestra.run.smithi163.stdout:(124/145): elfutils-libelf-devel-0.189-3.el8.x8 493 kB/s | 62 kB 00:00 2024-01-24T13:26:49.578 INFO:teuthology.orchestra.run.smithi163.stdout:(125/145): xfsprogs-devel-5.0.0-12.el8.x86_64.r 395 kB/s | 73 kB 00:00 2024-01-24T13:26:49.587 INFO:teuthology.orchestra.run.smithi029.stdout:(104/145): python3-oauthlib-2.1.0-1.el8.noarch. 797 kB/s | 155 kB 00:00 2024-01-24T13:26:49.619 INFO:teuthology.orchestra.run.smithi029.stdout:(105/145): python3-pycparser-2.14-14.el8.noarch 815 kB/s | 109 kB 00:00 2024-01-24T13:26:49.764 INFO:teuthology.orchestra.run.smithi163.stdout:(126/145): krb5-devel-1.18.2-26.el8_9.x86_64.rp 2.4 MB/s | 562 kB 00:00 2024-01-24T13:26:49.787 INFO:teuthology.orchestra.run.smithi163.stdout:(127/145): librabbitmq-0.9.0-4.el8.x86_64.rpm 144 kB/s | 47 kB 00:00 2024-01-24T13:26:49.854 INFO:teuthology.orchestra.run.smithi163.stdout:(128/145): openssl-devel-1.1.1k-12.el8_9.x86_64 8.4 MB/s | 2.3 MB 00:00 2024-01-24T13:26:49.903 INFO:teuthology.orchestra.run.smithi029.stdout:(106/145): python3-scipy-1.0.0-21.module+el8.9. 15 MB/s | 14 MB 00:00 2024-01-24T13:26:49.921 INFO:teuthology.orchestra.run.smithi163.stdout:(129/145): NetworkManager-initscripts-updown-1. 916 kB/s | 144 kB 00:00 2024-01-24T13:26:49.936 INFO:teuthology.orchestra.run.smithi029.stdout:(107/145): libaio-devel-0.3.112-1.el8.x86_64.rp 59 kB/s | 19 kB 00:00 2024-01-24T13:26:49.970 INFO:teuthology.orchestra.run.smithi029.stdout:(108/145): python3-requests-oauthlib-1.0.0-1.el 111 kB/s | 43 kB 00:00 2024-01-24T13:26:50.037 INFO:teuthology.orchestra.run.smithi029.stdout:(109/145): libzstd-devel-1.4.4-1.el8.x86_64.rpm 327 kB/s | 44 kB 00:00 2024-01-24T13:26:50.055 INFO:teuthology.orchestra.run.smithi163.stdout:(130/145): elfutils-libelf-0.189-3.el8.x86_64.r 1.7 MB/s | 232 kB 00:00 2024-01-24T13:26:50.079 INFO:teuthology.orchestra.run.smithi029.stdout:(110/145): pcre2-devel-10.32-2.el8.x86_64.rpm 4.1 MB/s | 605 kB 00:00 2024-01-24T13:26:50.105 INFO:teuthology.orchestra.run.smithi029.stdout:(111/145): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.7 MB/s | 229 kB 00:00 2024-01-24T13:26:50.164 INFO:teuthology.orchestra.run.smithi029.stdout:(112/145): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.7 MB/s | 220 kB 00:00 2024-01-24T13:26:50.199 INFO:teuthology.orchestra.run.smithi029.stdout:(113/145): keyutils-libs-devel-1.5.10-9.el8.x86 511 kB/s | 48 kB 00:00 2024-01-24T13:26:50.223 INFO:teuthology.orchestra.run.smithi163.stdout:(131/145): elfutils-libs-0.189-3.el8.x86_64.rpm 1.8 MB/s | 303 kB 00:00 2024-01-24T13:26:50.250 INFO:teuthology.orchestra.run.smithi029.stdout:(114/145): libselinux-devel-2.9-5.el8.x86_64.rp 1.1 MB/s | 200 kB 00:00 2024-01-24T13:26:50.275 INFO:teuthology.orchestra.run.smithi029.stdout:(115/145): zlib-devel-1.2.11-18.el8_5.x86_64.rp 762 kB/s | 58 kB 00:00 2024-01-24T13:26:50.301 INFO:teuthology.orchestra.run.smithi029.stdout:(116/145): libsepol-devel-2.9-3.el8.x86_64.rpm 636 kB/s | 87 kB 00:00 2024-01-24T13:26:50.364 INFO:teuthology.orchestra.run.smithi163.stdout:(132/145): librbd1-18.2.1-341.g4e3182a6.el8.x86 7.0 MB/s | 3.6 MB 00:00 2024-01-24T13:26:50.410 INFO:teuthology.orchestra.run.smithi029.stdout:(117/145): libcom_err-devel-1.45.6-4.el8.x86_64 242 kB/s | 39 kB 00:00 2024-01-24T13:26:50.423 INFO:teuthology.orchestra.run.smithi163.stdout:(133/145): librados2-18.2.1-341.g4e3182a6.el8.x 5.8 MB/s | 3.7 MB 00:00 2024-01-24T13:26:50.435 INFO:teuthology.orchestra.run.smithi029.stdout:(118/145): fuse-2.9.7-17.el8.x86_64.rpm 520 kB/s | 83 kB 00:00 2024-01-24T13:26:50.447 INFO:teuthology.orchestra.run.smithi163.stdout:(134/145): libmount-2.32.1-43.el8.x86_64.rpm 1.0 MB/s | 237 kB 00:00 2024-01-24T13:26:50.478 INFO:teuthology.orchestra.run.smithi029.stdout:(119/145): libgfortran-8.5.0-20.el8.x86_64.rpm 3.6 MB/s | 645 kB 00:00 2024-01-24T13:26:50.489 INFO:teuthology.orchestra.run.smithi163.stdout:(135/145): libsmartcols-2.32.1-43.el8.x86_64.rp 1.4 MB/s | 179 kB 00:00 2024-01-24T13:26:50.536 INFO:teuthology.orchestra.run.smithi029.stdout:(120/145): libquadmath-8.5.0-20.el8.x86_64.rpm 1.3 MB/s | 172 kB 00:00 2024-01-24T13:26:50.629 INFO:teuthology.orchestra.run.smithi029.stdout:(121/145): libuuid-devel-2.32.1-43.el8.x86_64.r 517 kB/s | 100 kB 00:00 2024-01-24T13:26:50.631 INFO:teuthology.orchestra.run.smithi163.stdout:(136/145): util-linux-2.32.1-43.el8.x86_64.rpm 14 MB/s | 2.5 MB 00:00 2024-01-24T13:26:50.657 INFO:teuthology.orchestra.run.smithi163.stdout:(137/145): libuuid-2.32.1-43.el8.x86_64.rpm 423 kB/s | 99 kB 00:00 2024-01-24T13:26:50.671 INFO:teuthology.orchestra.run.smithi029.stdout:(122/145): python3-cryptography-3.2.1-6.el8.x86 4.1 MB/s | 559 kB 00:00 2024-01-24T13:26:50.673 INFO:teuthology.orchestra.run.smithi163.stdout:(138/145): util-linux-user-2.32.1-43.el8.x86_64 559 kB/s | 103 kB 00:00 2024-01-24T13:26:50.738 INFO:teuthology.orchestra.run.smithi029.stdout:(123/145): xfsprogs-devel-5.0.0-12.el8.x86_64.r 666 kB/s | 73 kB 00:00 2024-01-24T13:26:50.780 INFO:teuthology.orchestra.run.smithi029.stdout:(124/145): python3-cffi-1.11.5-6.el8.x86_64.rpm 787 kB/s | 238 kB 00:00 2024-01-24T13:26:50.790 INFO:teuthology.orchestra.run.smithi163.stdout:(139/145): iproute-6.2.0-5.el8_9.x86_64.rpm 5.3 MB/s | 854 kB 00:00 2024-01-24T13:26:50.805 INFO:teuthology.orchestra.run.smithi029.stdout:(125/145): elfutils-libelf-devel-0.189-3.el8.x8 458 kB/s | 62 kB 00:00 2024-01-24T13:26:50.816 INFO:teuthology.orchestra.run.smithi163.stdout:(140/145): libfdisk-2.32.1-43.el8.x86_64.rpm 1.7 MB/s | 254 kB 00:00 2024-01-24T13:26:50.881 INFO:teuthology.orchestra.run.smithi029.stdout:(126/145): krb5-devel-1.18.2-26.el8_9.x86_64.rp 5.5 MB/s | 562 kB 00:00 2024-01-24T13:26:50.887 INFO:teuthology.orchestra.run.smithi163.stdout:(141/145): libblkid-2.32.1-43.el8.x86_64.rpm 959 kB/s | 220 kB 00:00 2024-01-24T13:26:50.906 INFO:teuthology.orchestra.run.smithi029.stdout:(127/145): librabbitmq-0.9.0-4.el8.x86_64.rpm 279 kB/s | 47 kB 00:00 2024-01-24T13:26:50.933 INFO:teuthology.orchestra.run.smithi163.stdout:(142/145): xfsprogs-5.0.0-12.el8.x86_64.rpm 7.5 MB/s | 1.1 MB 00:00 2024-01-24T13:26:50.966 INFO:teuthology.orchestra.run.smithi163.stdout:(143/145): openssl-1.1.1k-12.el8_9.x86_64.rpm 4.6 MB/s | 711 kB 00:00 2024-01-24T13:26:51.008 INFO:teuthology.orchestra.run.smithi163.stdout:(144/145): openssl-libs-1.1.1k-12.el8_9.x86_64. 12 MB/s | 1.5 MB 00:00 2024-01-24T13:26:51.067 INFO:teuthology.orchestra.run.smithi163.stdout:(145/145): xfsdump-3.1.8-7.el8_9.x86_64.rpm 2.4 MB/s | 333 kB 00:00 2024-01-24T13:26:51.068 INFO:teuthology.orchestra.run.smithi029.stdout:(128/145): openssl-devel-1.1.1k-12.el8_9.x86_64 8.9 MB/s | 2.3 MB 00:00 2024-01-24T13:26:51.069 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-01-24T13:26:51.070 INFO:teuthology.orchestra.run.smithi163.stdout:Total 11 MB/s | 212 MB 00:19 2024-01-24T13:26:51.101 INFO:teuthology.orchestra.run.smithi029.stdout:(129/145): NetworkManager-initscripts-updown-1. 656 kB/s | 144 kB 00:00 2024-01-24T13:26:51.235 INFO:teuthology.orchestra.run.smithi029.stdout:(130/145): elfutils-libelf-0.189-3.el8.x86_64.r 1.7 MB/s | 232 kB 00:00 2024-01-24T13:26:51.369 INFO:teuthology.orchestra.run.smithi029.stdout:(131/145): elfutils-libs-0.189-3.el8.x86_64.rpm 2.2 MB/s | 303 kB 00:00 2024-01-24T13:26:51.461 INFO:teuthology.orchestra.run.smithi029.stdout:(132/145): librbd1-18.2.1-341.g4e3182a6.el8.x86 9.1 MB/s | 3.6 MB 00:00 2024-01-24T13:26:51.495 INFO:teuthology.orchestra.run.smithi029.stdout:(133/145): libmount-2.32.1-43.el8.x86_64.rpm 1.8 MB/s | 237 kB 00:00 2024-01-24T13:26:51.553 INFO:teuthology.orchestra.run.smithi029.stdout:(134/145): libsmartcols-2.32.1-43.el8.x86_64.rp 1.9 MB/s | 179 kB 00:00 2024-01-24T13:26:51.604 INFO:teuthology.orchestra.run.smithi029.stdout:(135/145): libuuid-2.32.1-43.el8.x86_64.rpm 909 kB/s | 99 kB 00:00 2024-01-24T13:26:51.721 INFO:teuthology.orchestra.run.smithi029.stdout:(136/145): util-linux-2.32.1-43.el8.x86_64.rpm 15 MB/s | 2.5 MB 00:00 2024-01-24T13:26:51.830 INFO:teuthology.orchestra.run.smithi029.stdout:(137/145): util-linux-user-2.32.1-43.el8.x86_64 455 kB/s | 103 kB 00:00 2024-01-24T13:26:51.881 INFO:teuthology.orchestra.run.smithi029.stdout:(138/145): libblkid-2.32.1-43.el8.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-01-24T13:26:51.989 INFO:teuthology.orchestra.run.smithi029.stdout:(139/145): libfdisk-2.32.1-43.el8.x86_64.rpm 2.3 MB/s | 254 kB 00:00 2024-01-24T13:26:52.065 INFO:teuthology.orchestra.run.smithi029.stdout:(140/145): librados2-18.2.1-341.g4e3182a6.el8.x 3.2 MB/s | 3.7 MB 00:01 2024-01-24T13:26:52.095 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-01-24T13:26:52.107 INFO:teuthology.orchestra.run.smithi029.stdout:(141/145): iproute-6.2.0-5.el8_9.x86_64.rpm 2.2 MB/s | 854 kB 00:00 2024-01-24T13:26:52.158 INFO:teuthology.orchestra.run.smithi029.stdout:(142/145): xfsprogs-5.0.0-12.el8.x86_64.rpm 6.4 MB/s | 1.1 MB 00:00 2024-01-24T13:26:52.208 INFO:teuthology.orchestra.run.smithi029.stdout:(143/145): openssl-1.1.1k-12.el8_9.x86_64.rpm 4.9 MB/s | 711 kB 00:00 2024-01-24T13:26:52.267 INFO:teuthology.orchestra.run.smithi029.stdout:(144/145): openssl-libs-1.1.1k-12.el8_9.x86_64. 9.2 MB/s | 1.5 MB 00:00 2024-01-24T13:26:52.294 INFO:teuthology.orchestra.run.smithi029.stdout:(145/145): xfsdump-3.1.8-7.el8_9.x86_64.rpm 2.4 MB/s | 333 kB 00:00 2024-01-24T13:26:52.299 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-01-24T13:26:52.299 INFO:teuthology.orchestra.run.smithi029.stdout:Total 12 MB/s | 212 MB 00:17 2024-01-24T13:26:52.489 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-01-24T13:26:52.489 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-01-24T13:26:53.492 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-01-24T13:26:53.680 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-01-24T13:26:53.691 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-01-24T13:26:53.942 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-01-24T13:26:53.942 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-01-24T13:26:54.048 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-01-24T13:26:54.182 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/1 2024-01-24T13:26:54.423 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/161 2024-01-24T13:26:54.633 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/161 2024-01-24T13:26:54.840 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/161 2024-01-24T13:26:55.017 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : libuuid-2.32.1-43.el8.x86_64 3/161 2024-01-24T13:26:55.078 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 3/161 2024-01-24T13:26:55.251 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : libblkid-2.32.1-43.el8.x86_64 4/161 2024-01-24T13:26:55.254 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-01-24T13:26:55.266 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-01-24T13:26:55.340 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libblkid-2.32.1-43.el8.x86_64 4/161 2024-01-24T13:26:55.557 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/161 2024-01-24T13:26:55.734 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 6/161 2024-01-24T13:26:55.798 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 6/161 2024-01-24T13:26:56.020 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/161 2024-01-24T13:26:56.192 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 8/161 2024-01-24T13:26:56.595 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 8/161 2024-01-24T13:26:56.933 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : librados2-2:18.2.1-341.g4e3182a6.el8.x86_64 9/161 2024-01-24T13:26:57.329 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librados2-2:18.2.1-341.g4e3182a6.el8.x86_64 9/161 2024-01-24T13:26:57.525 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : librbd1-2:18.2.1-341.g4e3182a6.el8.x86_64 10/161 2024-01-24T13:26:57.638 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librbd1-2:18.2.1-341.g4e3182a6.el8.x86_64 10/161 2024-01-24T13:26:57.846 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libcephfs2-2:18.2.1-341.g4e3182a6.el8.x86_64 11/161 2024-01-24T13:26:57.942 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libcephfs2-2:18.2.1-341.g4e3182a6.el8.x86_64 11/161 2024-01-24T13:26:58.260 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rados-2:18.2.1-341.g4e3182a6.el8.x86_64 12/161 2024-01-24T13:26:58.575 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 13/161 2024-01-24T13:26:58.658 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 13/161 2024-01-24T13:26:58.856 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libpq-13.11-1.el8.x86_64 14/161 2024-01-24T13:26:59.112 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : elfutils-libelf-0.189-3.el8.x86_64 15/161 2024-01-24T13:26:59.295 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 16/161 2024-01-24T13:26:59.687 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 17/161 2024-01-24T13:26:59.761 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-01-24T13:26:59.896 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/1 2024-01-24T13:26:59.945 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : postgresql-10.23-3.module+el8.9.0+20831+78489779 18/161 2024-01-24T13:27:00.115 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libcephsqlite-2:18.2.1-341.g4e3182a6.el8.x86_64 19/161 2024-01-24T13:27:00.153 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/161 2024-01-24T13:27:00.218 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libcephsqlite-2:18.2.1-341.g4e3182a6.el8.x86_64 19/161 2024-01-24T13:27:00.366 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libradosstriper1-2:18.2.1-341.g4e3182a6.el8.x86_ 20/161 2024-01-24T13:27:00.390 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/161 2024-01-24T13:27:00.465 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libradosstriper1-2:18.2.1-341.g4e3182a6.el8.x86_ 20/161 2024-01-24T13:27:00.624 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 21/161 2024-01-24T13:27:00.639 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/161 2024-01-24T13:27:00.757 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 21/161 2024-01-24T13:27:00.854 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : libuuid-2.32.1-43.el8.x86_64 3/161 2024-01-24T13:27:00.916 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 22/161 2024-01-24T13:27:00.926 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 3/161 2024-01-24T13:27:01.088 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : libblkid-2.32.1-43.el8.x86_64 4/161 2024-01-24T13:27:01.187 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libblkid-2.32.1-43.el8.x86_64 4/161 2024-01-24T13:27:01.297 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-01-24T13:27:01.411 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/161 2024-01-24T13:27:01.447 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/1 2024-01-24T13:27:01.542 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 22/161 2024-01-24T13:27:01.638 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 6/161 2024-01-24T13:27:01.699 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 23/161 2024-01-24T13:27:01.704 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 6/161 2024-01-24T13:27:01.722 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/161 2024-01-24T13:27:01.908 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/161 2024-01-24T13:27:01.963 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/161 2024-01-24T13:27:02.104 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 8/161 2024-01-24T13:27:02.150 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/161 2024-01-24T13:27:02.305 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 23/161 2024-01-24T13:27:02.339 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : libuuid-2.32.1-43.el8.x86_64 3/161 2024-01-24T13:27:02.420 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 3/161 2024-01-24T13:27:02.508 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : openblas-0.3.15-6.el8.x86_64 24/161 2024-01-24T13:27:02.511 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 8/161 2024-01-24T13:27:02.606 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : libblkid-2.32.1-43.el8.x86_64 4/161 2024-01-24T13:27:02.681 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libblkid-2.32.1-43.el8.x86_64 4/161 2024-01-24T13:27:02.746 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : librados2-2:18.2.1-341.g4e3182a6.el8.x86_64 9/161 2024-01-24T13:27:02.950 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/161 2024-01-24T13:27:03.145 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librados2-2:18.2.1-341.g4e3182a6.el8.x86_64 9/161 2024-01-24T13:27:03.148 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 6/161 2024-01-24T13:27:03.196 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 24/161 2024-01-24T13:27:03.225 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 6/161 2024-01-24T13:27:03.296 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : librbd1-2:18.2.1-341.g4e3182a6.el8.x86_64 10/161 2024-01-24T13:27:03.431 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/161 2024-01-24T13:27:03.436 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librbd1-2:18.2.1-341.g4e3182a6.el8.x86_64 10/161 2024-01-24T13:27:03.480 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 25/161 2024-01-24T13:27:03.629 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 26/161 2024-01-24T13:27:03.648 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 8/161 2024-01-24T13:27:03.663 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephfs2-2:18.2.1-341.g4e3182a6.el8.x86_64 11/161 2024-01-24T13:27:03.740 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libcephfs2-2:18.2.1-341.g4e3182a6.el8.x86_64 11/161 2024-01-24T13:27:03.786 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 27/161 2024-01-24T13:27:04.066 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/161 2024-01-24T13:27:04.073 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rados-2:18.2.1-341.g4e3182a6.el8.x86_64 12/161 2024-01-24T13:27:04.094 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 8/161 2024-01-24T13:27:04.338 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 29/161 2024-01-24T13:27:04.355 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librados2-2:18.2.1-341.g4e3182a6.el8.x86_64 9/161 2024-01-24T13:27:04.362 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 13/161 2024-01-24T13:27:04.448 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 13/161 2024-01-24T13:27:04.617 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 30/161 2024-01-24T13:27:04.669 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libpq-13.11-1.el8.x86_64 14/161 2024-01-24T13:27:04.793 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-ceph-common-2:18.2.1-341.g4e3182a6.el8.x 31/161 2024-01-24T13:27:04.797 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-2:18.2.1-341.g4e3182a6.el8.x86_64 9/161 2024-01-24T13:27:04.926 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-ceph-argparse-2:18.2.1-341.g4e3182a6.el8 32/161 2024-01-24T13:27:04.943 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : elfutils-libelf-0.189-3.el8.x86_64 15/161 2024-01-24T13:27:04.964 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librbd1-2:18.2.1-341.g4e3182a6.el8.x86_64 10/161 2024-01-24T13:27:05.103 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-2:18.2.1-341.g4e3182a6.el8.x86_64 10/161 2024-01-24T13:27:05.108 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 16/161 2024-01-24T13:27:05.184 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cephfs-2:18.2.1-341.g4e3182a6.el8.x86_64 33/161 2024-01-24T13:27:05.297 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs2-2:18.2.1-341.g4e3182a6.el8.x86_64 11/161 2024-01-24T13:27:05.408 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephfs2-2:18.2.1-341.g4e3182a6.el8.x86_64 11/161 2024-01-24T13:27:05.510 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 17/161 2024-01-24T13:27:05.743 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rados-2:18.2.1-341.g4e3182a6.el8.x86_64 12/161 2024-01-24T13:27:06.008 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : postgresql-10.23-3.module+el8.9.0+20831+78489779 18/161 2024-01-24T13:27:06.039 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 13/161 2024-01-24T13:27:06.127 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 34/161 2024-01-24T13:27:06.133 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 13/161 2024-01-24T13:27:06.229 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephsqlite-2:18.2.1-341.g4e3182a6.el8.x86_64 19/161 2024-01-24T13:27:06.316 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libcephsqlite-2:18.2.1-341.g4e3182a6.el8.x86_64 19/161 2024-01-24T13:27:06.341 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libpq-13.11-1.el8.x86_64 14/161 2024-01-24T13:27:06.462 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 35/161 2024-01-24T13:27:06.504 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libradosstriper1-2:18.2.1-341.g4e3182a6.el8.x86_ 20/161 2024-01-24T13:27:06.579 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libradosstriper1-2:18.2.1-341.g4e3182a6.el8.x86_ 20/161 2024-01-24T13:27:06.597 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : elfutils-libelf-0.189-3.el8.x86_64 15/161 2024-01-24T13:27:06.762 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 21/161 2024-01-24T13:27:06.762 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 16/161 2024-01-24T13:27:06.770 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 36/161 2024-01-24T13:27:06.896 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 21/161 2024-01-24T13:27:06.948 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 37/161 2024-01-24T13:27:07.062 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 22/161 2024-01-24T13:27:07.158 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 17/161 2024-01-24T13:27:07.445 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : postgresql-10.23-3.module+el8.9.0+20831+78489779 18/161 2024-01-24T13:27:07.611 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephsqlite-2:18.2.1-341.g4e3182a6.el8.x86_64 19/161 2024-01-24T13:27:07.691 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 22/161 2024-01-24T13:27:07.729 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephsqlite-2:18.2.1-341.g4e3182a6.el8.x86_64 19/161 2024-01-24T13:27:07.878 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 23/161 2024-01-24T13:27:07.881 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libradosstriper1-2:18.2.1-341.g4e3182a6.el8.x86_ 20/161 2024-01-24T13:27:07.965 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libradosstriper1-2:18.2.1-341.g4e3182a6.el8.x86_ 20/161 2024-01-24T13:27:08.130 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 21/161 2024-01-24T13:27:08.276 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 21/161 2024-01-24T13:27:08.405 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 22/161 2024-01-24T13:27:08.511 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 23/161 2024-01-24T13:27:08.685 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 37/161 2024-01-24T13:27:08.720 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : openblas-0.3.15-6.el8.x86_64 24/161 2024-01-24T13:27:08.990 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 38/161 2024-01-24T13:27:09.096 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 22/161 2024-01-24T13:27:09.226 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rsa-4.9-2.el8.noarch 39/161 2024-01-24T13:27:09.264 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 23/161 2024-01-24T13:27:09.410 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 24/161 2024-01-24T13:27:09.441 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 40/161 2024-01-24T13:27:09.653 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rbd-2:18.2.1-341.g4e3182a6.el8.x86_64 41/161 2024-01-24T13:27:09.693 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 25/161 2024-01-24T13:27:09.845 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librados-devel-2:18.2.1-341.g4e3182a6.el8.x86_64 42/161 2024-01-24T13:27:09.875 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 26/161 2024-01-24T13:27:09.937 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 23/161 2024-01-24T13:27:10.023 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 27/161 2024-01-24T13:27:10.089 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : libmount-2.32.1-43.el8.x86_64 43/161 2024-01-24T13:27:10.123 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-0.3.15-6.el8.x86_64 24/161 2024-01-24T13:27:10.166 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libmount-2.32.1-43.el8.x86_64 43/161 2024-01-24T13:27:10.321 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/161 2024-01-24T13:27:10.398 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : libfdisk-2.32.1-43.el8.x86_64 44/161 2024-01-24T13:27:10.467 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libfdisk-2.32.1-43.el8.x86_64 44/161 2024-01-24T13:27:10.626 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 29/161 2024-01-24T13:27:10.711 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libuuid-devel-2.32.1-43.el8.x86_64 45/161 2024-01-24T13:27:10.913 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 30/161 2024-01-24T13:27:10.930 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 24/161 2024-01-24T13:27:10.946 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 46/161 2024-01-24T13:27:11.139 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : libsmartcols-2.32.1-43.el8.x86_64 47/161 2024-01-24T13:27:11.140 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-ceph-common-2:18.2.1-341.g4e3182a6.el8.x 31/161 2024-01-24T13:27:11.221 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 25/161 2024-01-24T13:27:11.280 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-ceph-argparse-2:18.2.1-341.g4e3182a6.el8 32/161 2024-01-24T13:27:11.373 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 26/161 2024-01-24T13:27:11.556 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cephfs-2:18.2.1-341.g4e3182a6.el8.x86_64 33/161 2024-01-24T13:27:11.562 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 27/161 2024-01-24T13:27:11.596 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libsmartcols-2.32.1-43.el8.x86_64 47/161 2024-01-24T13:27:11.880 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/161 2024-01-24T13:27:11.915 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : util-linux-2.32.1-43.el8.x86_64 48/161 2024-01-24T13:27:11.932 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: util-linux-2.32.1-43.el8.x86_64 48/161 2024-01-24T13:27:12.190 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 29/161 2024-01-24T13:27:12.442 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 30/161 2024-01-24T13:27:12.503 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 34/161 2024-01-24T13:27:12.519 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : fuse-2.9.7-17.el8.x86_64 49/161 2024-01-24T13:27:12.633 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-common-2:18.2.1-341.g4e3182a6.el8.x 31/161 2024-01-24T13:27:12.746 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 50/161 2024-01-24T13:27:12.769 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-argparse-2:18.2.1-341.g4e3182a6.el8 32/161 2024-01-24T13:27:12.850 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 35/161 2024-01-24T13:27:12.911 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 51/161 2024-01-24T13:27:13.029 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cephfs-2:18.2.1-341.g4e3182a6.el8.x86_64 33/161 2024-01-24T13:27:13.133 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 52/161 2024-01-24T13:27:13.148 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 36/161 2024-01-24T13:27:13.343 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 53/161 2024-01-24T13:27:13.378 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 37/161 2024-01-24T13:27:13.628 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 54/161 2024-01-24T13:27:13.909 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 55/161 2024-01-24T13:27:14.121 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 34/161 2024-01-24T13:27:14.135 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 56/161 2024-01-24T13:27:14.318 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 57/161 2024-01-24T13:27:14.466 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 35/161 2024-01-24T13:27:14.549 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 58/161 2024-01-24T13:27:14.780 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 36/161 2024-01-24T13:27:14.944 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 59/161 2024-01-24T13:27:14.980 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 37/161 2024-01-24T13:27:15.125 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 37/161 2024-01-24T13:27:15.250 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 60/161 2024-01-24T13:27:15.415 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 61/161 2024-01-24T13:27:15.470 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 38/161 2024-01-24T13:27:15.617 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 62/161 2024-01-24T13:27:15.681 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rsa-4.9-2.el8.noarch 39/161 2024-01-24T13:27:15.884 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 63/161 2024-01-24T13:27:15.905 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 40/161 2024-01-24T13:27:16.133 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rbd-2:18.2.1-341.g4e3182a6.el8.x86_64 41/161 2024-01-24T13:27:16.221 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 64/161 2024-01-24T13:27:16.325 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librados-devel-2:18.2.1-341.g4e3182a6.el8.x86_64 42/161 2024-01-24T13:27:16.438 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 65/161 2024-01-24T13:27:16.552 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : libmount-2.32.1-43.el8.x86_64 43/161 2024-01-24T13:27:16.610 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 66/161 2024-01-24T13:27:16.654 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libmount-2.32.1-43.el8.x86_64 43/161 2024-01-24T13:27:16.841 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 67/161 2024-01-24T13:27:16.869 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : libfdisk-2.32.1-43.el8.x86_64 44/161 2024-01-24T13:27:16.957 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 37/161 2024-01-24T13:27:16.962 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libfdisk-2.32.1-43.el8.x86_64 44/161 2024-01-24T13:27:17.043 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 68/161 2024-01-24T13:27:17.189 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libuuid-devel-2.32.1-43.el8.x86_64 45/161 2024-01-24T13:27:17.305 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 38/161 2024-01-24T13:27:17.390 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : krb5-devel-1.18.2-26.el8_9.x86_64 69/161 2024-01-24T13:27:17.425 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 46/161 2024-01-24T13:27:17.597 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : automake-1.16.1-8.el8.noarch 70/161 2024-01-24T13:27:17.600 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rsa-4.9-2.el8.noarch 39/161 2024-01-24T13:27:17.685 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : libsmartcols-2.32.1-43.el8.x86_64 47/161 2024-01-24T13:27:17.846 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : uuid-1.6.2-43.el8.x86_64 71/161 2024-01-24T13:27:17.853 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 40/161 2024-01-24T13:27:17.927 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 71/161 2024-01-24T13:27:18.054 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rbd-2:18.2.1-341.g4e3182a6.el8.x86_64 41/161 2024-01-24T13:27:18.104 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 72/161 2024-01-24T13:27:18.131 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libsmartcols-2.32.1-43.el8.x86_64 47/161 2024-01-24T13:27:18.184 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 72/161 2024-01-24T13:27:18.230 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librados-devel-2:18.2.1-341.g4e3182a6.el8.x86_64 42/161 2024-01-24T13:27:18.431 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : jq-1.6-7.el8.x86_64 73/161 2024-01-24T13:27:18.436 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : libmount-2.32.1-43.el8.x86_64 43/161 2024-01-24T13:27:18.469 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : util-linux-2.32.1-43.el8.x86_64 48/161 2024-01-24T13:27:18.485 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: util-linux-2.32.1-43.el8.x86_64 48/161 2024-01-24T13:27:18.516 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libmount-2.32.1-43.el8.x86_64 43/161 2024-01-24T13:27:18.620 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 74/161 2024-01-24T13:27:18.705 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : fuse-2.9.7-17.el8.x86_64 49/161 2024-01-24T13:27:18.754 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : libfdisk-2.32.1-43.el8.x86_64 44/161 2024-01-24T13:27:18.826 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libfdisk-2.32.1-43.el8.x86_64 44/161 2024-01-24T13:27:18.839 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 75/161 2024-01-24T13:27:18.930 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 50/161 2024-01-24T13:27:19.014 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 76/161 2024-01-24T13:27:19.098 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libuuid-devel-2.32.1-43.el8.x86_64 45/161 2024-01-24T13:27:19.139 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 51/161 2024-01-24T13:27:19.171 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 77/161 2024-01-24T13:27:19.311 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 52/161 2024-01-24T13:27:19.333 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 46/161 2024-01-24T13:27:19.385 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 78/161 2024-01-24T13:27:19.521 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : libsmartcols-2.32.1-43.el8.x86_64 47/161 2024-01-24T13:27:19.530 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 53/161 2024-01-24T13:27:19.669 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 79/161 2024-01-24T13:27:19.827 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 54/161 2024-01-24T13:27:19.928 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 80/161 2024-01-24T13:27:20.010 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libsmartcols-2.32.1-43.el8.x86_64 47/161 2024-01-24T13:27:20.115 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 55/161 2024-01-24T13:27:20.134 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 81/161 2024-01-24T13:27:20.299 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 82/161 2024-01-24T13:27:20.304 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : util-linux-2.32.1-43.el8.x86_64 48/161 2024-01-24T13:27:20.326 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: util-linux-2.32.1-43.el8.x86_64 48/161 2024-01-24T13:27:20.340 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 56/161 2024-01-24T13:27:20.514 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 57/161 2024-01-24T13:27:20.539 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 83/161 2024-01-24T13:27:20.585 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : fuse-2.9.7-17.el8.x86_64 49/161 2024-01-24T13:27:20.767 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 84/161 2024-01-24T13:27:20.771 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 58/161 2024-01-24T13:27:20.773 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 50/161 2024-01-24T13:27:20.962 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 51/161 2024-01-24T13:27:20.994 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 85/161 2024-01-24T13:27:21.158 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 59/161 2024-01-24T13:27:21.167 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 52/161 2024-01-24T13:27:21.191 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 86/161 2024-01-24T13:27:21.369 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 53/161 2024-01-24T13:27:21.376 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 87/161 2024-01-24T13:27:21.579 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 60/161 2024-01-24T13:27:21.600 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 88/161 2024-01-24T13:27:21.684 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 54/161 2024-01-24T13:27:21.752 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 61/161 2024-01-24T13:27:21.841 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 89/161 2024-01-24T13:27:21.971 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 55/161 2024-01-24T13:27:22.013 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 62/161 2024-01-24T13:27:22.043 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 90/161 2024-01-24T13:27:22.219 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 56/161 2024-01-24T13:27:22.289 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 63/161 2024-01-24T13:27:22.306 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-portend-2.6-1.el8.noarch 91/161 2024-01-24T13:27:22.398 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 57/161 2024-01-24T13:27:22.586 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 92/161 2024-01-24T13:27:22.642 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 64/161 2024-01-24T13:27:22.649 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 58/161 2024-01-24T13:27:22.769 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 93/161 2024-01-24T13:27:22.892 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 65/161 2024-01-24T13:27:23.037 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 94/161 2024-01-24T13:27:23.052 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 59/161 2024-01-24T13:27:23.055 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 66/161 2024-01-24T13:27:23.295 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 67/161 2024-01-24T13:27:23.362 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 60/161 2024-01-24T13:27:23.531 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 68/161 2024-01-24T13:27:23.619 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 61/161 2024-01-24T13:27:23.873 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 62/161 2024-01-24T13:27:23.904 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : krb5-devel-1.18.2-26.el8_9.x86_64 69/161 2024-01-24T13:27:24.088 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 95/161 2024-01-24T13:27:24.131 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 63/161 2024-01-24T13:27:24.168 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : automake-1.16.1-8.el8.noarch 70/161 2024-01-24T13:27:24.334 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : uuid-1.6.2-43.el8.x86_64 71/161 2024-01-24T13:27:24.398 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 96/161 2024-01-24T13:27:24.438 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 71/161 2024-01-24T13:27:24.502 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 64/161 2024-01-24T13:27:24.634 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 72/161 2024-01-24T13:27:24.652 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 97/161 2024-01-24T13:27:24.714 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 65/161 2024-01-24T13:27:24.737 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 72/161 2024-01-24T13:27:24.942 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : jq-1.6-7.el8.x86_64 73/161 2024-01-24T13:27:24.962 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 66/161 2024-01-24T13:27:24.968 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 98/161 2024-01-24T13:27:25.141 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 74/161 2024-01-24T13:27:25.165 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 67/161 2024-01-24T13:27:25.229 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 99/161 2024-01-24T13:27:25.350 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 75/161 2024-01-24T13:27:25.370 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 68/161 2024-01-24T13:27:25.436 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 100/161 2024-01-24T13:27:25.544 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 76/161 2024-01-24T13:27:25.726 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 77/161 2024-01-24T13:27:25.772 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : krb5-devel-1.18.2-26.el8_9.x86_64 69/161 2024-01-24T13:27:25.906 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 78/161 2024-01-24T13:27:25.965 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : automake-1.16.1-8.el8.noarch 70/161 2024-01-24T13:27:26.162 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : uuid-1.6.2-43.el8.x86_64 71/161 2024-01-24T13:27:26.215 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 79/161 2024-01-24T13:27:26.224 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 101/161 2024-01-24T13:27:26.236 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 71/161 2024-01-24T13:27:26.378 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librgw2-2:18.2.1-341.g4e3182a6.el8.x86_64 102/161 2024-01-24T13:27:26.475 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 80/161 2024-01-24T13:27:26.487 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 72/161 2024-01-24T13:27:26.487 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librgw2-2:18.2.1-341.g4e3182a6.el8.x86_64 102/161 2024-01-24T13:27:26.571 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 72/161 2024-01-24T13:27:26.687 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rgw-2:18.2.1-341.g4e3182a6.el8.x86_64 103/161 2024-01-24T13:27:26.714 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 81/161 2024-01-24T13:27:26.772 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : jq-1.6-7.el8.x86_64 73/161 2024-01-24T13:27:26.920 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 82/161 2024-01-24T13:27:26.962 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 74/161 2024-01-24T13:27:27.119 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 83/161 2024-01-24T13:27:27.137 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 75/161 2024-01-24T13:27:27.323 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 76/161 2024-01-24T13:27:27.374 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 84/161 2024-01-24T13:27:27.505 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 77/161 2024-01-24T13:27:27.640 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 85/161 2024-01-24T13:27:27.696 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 78/161 2024-01-24T13:27:27.820 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 86/161 2024-01-24T13:27:27.956 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 87/161 2024-01-24T13:27:28.051 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 79/161 2024-01-24T13:27:28.180 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 88/161 2024-01-24T13:27:28.303 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 80/161 2024-01-24T13:27:28.437 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 89/161 2024-01-24T13:27:28.517 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 81/161 2024-01-24T13:27:28.622 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 90/161 2024-01-24T13:27:28.691 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 82/161 2024-01-24T13:27:28.904 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-portend-2.6-1.el8.noarch 91/161 2024-01-24T13:27:28.917 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 83/161 2024-01-24T13:27:29.165 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 84/161 2024-01-24T13:27:29.166 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 92/161 2024-01-24T13:27:29.391 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 93/161 2024-01-24T13:27:29.458 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 85/161 2024-01-24T13:27:29.639 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 94/161 2024-01-24T13:27:29.657 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 86/161 2024-01-24T13:27:29.818 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 87/161 2024-01-24T13:27:30.047 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 88/161 2024-01-24T13:27:30.284 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 89/161 2024-01-24T13:27:30.484 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 90/161 2024-01-24T13:27:30.751 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 95/161 2024-01-24T13:27:30.843 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-common-2:18.2.1-341.g4e3182a6.el8.x86_64 104/161 2024-01-24T13:27:30.985 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-portend-2.6-1.el8.noarch 91/161 2024-01-24T13:27:31.095 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-common-2:18.2.1-341.g4e3182a6.el8.x86_64 104/161 2024-01-24T13:27:31.104 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 96/161 2024-01-24T13:27:31.365 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 97/161 2024-01-24T13:27:31.454 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 92/161 2024-01-24T13:27:31.653 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 93/161 2024-01-24T13:27:31.672 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-common-2:18.2.1-341.g4e3182a6.el8.x86_64 104/161 2024-01-24T13:27:31.684 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 98/161 2024-01-24T13:27:31.881 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 94/161 2024-01-24T13:27:31.951 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-base-2:18.2.1-341.g4e3182a6.el8.x86_64 105/161 2024-01-24T13:27:31.983 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 99/161 2024-01-24T13:27:32.024 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-base-2:18.2.1-341.g4e3182a6.el8.x86_64 105/161 2024-01-24T13:27:32.199 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 100/161 2024-01-24T13:27:32.210 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-selinux-2:18.2.1-341.g4e3182a6.el8.x86_64 106/161 2024-01-24T13:27:33.016 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 101/161 2024-01-24T13:27:33.094 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 95/161 2024-01-24T13:27:33.182 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librgw2-2:18.2.1-341.g4e3182a6.el8.x86_64 102/161 2024-01-24T13:27:33.239 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librgw2-2:18.2.1-341.g4e3182a6.el8.x86_64 102/161 2024-01-24T13:27:33.447 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 96/161 2024-01-24T13:27:33.458 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rgw-2:18.2.1-341.g4e3182a6.el8.x86_64 103/161 2024-01-24T13:27:33.707 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 97/161 2024-01-24T13:27:34.099 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 98/161 2024-01-24T13:27:34.364 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 99/161 2024-01-24T13:27:34.574 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 100/161 2024-01-24T13:27:35.413 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 101/161 2024-01-24T13:27:35.594 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librgw2-2:18.2.1-341.g4e3182a6.el8.x86_64 102/161 2024-01-24T13:27:35.688 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librgw2-2:18.2.1-341.g4e3182a6.el8.x86_64 102/161 2024-01-24T13:27:35.878 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rgw-2:18.2.1-341.g4e3182a6.el8.x86_64 103/161 2024-01-24T13:27:37.183 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-common-2:18.2.1-341.g4e3182a6.el8.x86_64 104/161 2024-01-24T13:27:38.083 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-common-2:18.2.1-341.g4e3182a6.el8.x86_64 104/161 2024-01-24T13:27:38.669 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-common-2:18.2.1-341.g4e3182a6.el8.x86_64 104/161 2024-01-24T13:27:38.972 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-base-2:18.2.1-341.g4e3182a6.el8.x86_64 105/161 2024-01-24T13:27:39.042 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-base-2:18.2.1-341.g4e3182a6.el8.x86_64 105/161 2024-01-24T13:27:39.232 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-selinux-2:18.2.1-341.g4e3182a6.el8.x86_64 106/161 2024-01-24T13:27:40.915 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:18.2.1-341.g4e3182a6.el8.x86_64 104/161 2024-01-24T13:27:41.161 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-common-2:18.2.1-341.g4e3182a6.el8.x86_64 104/161 2024-01-24T13:27:41.805 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:18.2.1-341.g4e3182a6.el8.x86_64 104/161 2024-01-24T13:27:42.101 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-base-2:18.2.1-341.g4e3182a6.el8.x86_64 105/161 2024-01-24T13:27:42.202 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-base-2:18.2.1-341.g4e3182a6.el8.x86_64 105/161 2024-01-24T13:27:42.352 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-selinux-2:18.2.1-341.g4e3182a6.el8.x86_64 106/161 2024-01-24T13:27:51.498 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-selinux-2:18.2.1-341.g4e3182a6.el8.x86_64 106/161 2024-01-24T13:27:51.498 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /sys 2024-01-24T13:27:51.498 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /proc 2024-01-24T13:27:51.498 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /dev 2024-01-24T13:27:51.498 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /run 2024-01-24T13:27:51.498 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /mnt 2024-01-24T13:27:51.498 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /var/tmp 2024-01-24T13:27:51.499 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /home 2024-01-24T13:27:51.499 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /tmp 2024-01-24T13:27:51.499 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /dev 2024-01-24T13:27:51.499 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-01-24T13:27:51.849 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-volume-2:18.2.1-341.g4e3182a6.el8.noarch 107/161 2024-01-24T13:27:53.415 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-volume-2:18.2.1-341.g4e3182a6.el8.noarch 107/161 2024-01-24T13:27:53.622 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-osd-2:18.2.1-341.g4e3182a6.el8.x86_64 108/161 2024-01-24T13:27:53.841 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-osd-2:18.2.1-341.g4e3182a6.el8.x86_64 108/161 2024-01-24T13:27:54.015 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mds-2:18.2.1-341.g4e3182a6.el8.x86_64 109/161 2024-01-24T13:27:54.470 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mds-2:18.2.1-341.g4e3182a6.el8.x86_64 109/161 2024-01-24T13:27:54.673 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mon-2:18.2.1-341.g4e3182a6.el8.x86_64 110/161 2024-01-24T13:27:54.691 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mon-2:18.2.1-341.g4e3182a6.el8.x86_64 110/161 2024-01-24T13:27:56.360 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: cephadm-2:18.2.1-341.g4e3182a6.el8.noarch 111/161 2024-01-24T13:27:56.515 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : cephadm-2:18.2.1-341.g4e3182a6.el8.noarch 111/161 2024-01-24T13:27:56.679 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-prometheus-alerts-2:18.2.1-341.g4e3182a6.el 112/161 2024-01-24T13:27:56.855 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-grafana-dashboards-2:18.2.1-341.g4e3182a6.e 113/161 2024-01-24T13:27:57.106 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-cephadm-2:18.2.1-341.g4e3182a6.el8.noar 114/161 2024-01-24T13:27:57.331 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.1-341.g4e3182a6.el8.noar 114/161 2024-01-24T13:27:57.654 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-dashboard-2:18.2.1-341.g4e3182a6.el8.no 115/161 2024-01-24T13:27:58.619 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.1-341.g4e3182a6.el8.no 115/161 2024-01-24T13:27:58.636 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-selinux-2:18.2.1-341.g4e3182a6.el8.x86_64 106/161 2024-01-24T13:27:58.636 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /sys 2024-01-24T13:27:58.637 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /proc 2024-01-24T13:27:58.637 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /dev 2024-01-24T13:27:58.637 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /run 2024-01-24T13:27:58.637 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /mnt 2024-01-24T13:27:58.637 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /var/tmp 2024-01-24T13:27:58.637 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /home 2024-01-24T13:27:58.637 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /tmp 2024-01-24T13:27:58.637 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /dev 2024-01-24T13:27:58.637 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-01-24T13:27:58.797 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.1-341.g4e31 116/161 2024-01-24T13:27:58.890 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.1-341.g4e31 116/161 2024-01-24T13:27:59.029 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-volume-2:18.2.1-341.g4e3182a6.el8.noarch 107/161 2024-01-24T13:27:59.272 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-modules-core-2:18.2.1-341.g4e3182a6.el8 117/161 2024-01-24T13:27:59.446 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-2:18.2.1-341.g4e3182a6.el8.x86_64 118/161 2024-01-24T13:27:59.484 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-2:18.2.1-341.g4e3182a6.el8.x86_64 118/161 2024-01-24T13:27:59.664 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-rook-2:18.2.1-341.g4e3182a6.el8.noarch 119/161 2024-01-24T13:27:59.676 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.1-341.g4e3182a6.el8.noarch 119/161 2024-01-24T13:28:00.601 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-volume-2:18.2.1-341.g4e3182a6.el8.noarch 107/161 2024-01-24T13:28:00.793 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-osd-2:18.2.1-341.g4e3182a6.el8.x86_64 108/161 2024-01-24T13:28:00.937 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-2:18.2.1-341.g4e3182a6.el8.x86_64 120/161 2024-01-24T13:28:01.014 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-osd-2:18.2.1-341.g4e3182a6.el8.x86_64 108/161 2024-01-24T13:28:01.187 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-radosgw-2:18.2.1-341.g4e3182a6.el8.x86_64 121/161 2024-01-24T13:28:01.203 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mds-2:18.2.1-341.g4e3182a6.el8.x86_64 109/161 2024-01-24T13:28:01.222 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-radosgw-2:18.2.1-341.g4e3182a6.el8.x86_64 121/161 2024-01-24T13:28:01.397 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-immutable-object-cache-2:18.2.1-341.g4e3182 122/161 2024-01-24T13:28:01.441 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.1-341.g4e3182 122/161 2024-01-24T13:28:01.606 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : cephfs-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 123/161 2024-01-24T13:28:01.662 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mds-2:18.2.1-341.g4e3182a6.el8.x86_64 109/161 2024-01-24T13:28:01.861 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mon-2:18.2.1-341.g4e3182a6.el8.x86_64 110/161 2024-01-24T13:28:01.879 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mon-2:18.2.1-341.g4e3182a6.el8.x86_64 110/161 2024-01-24T13:28:01.952 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: cephfs-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 123/161 2024-01-24T13:28:02.113 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : rbd-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 124/161 2024-01-24T13:28:03.601 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: cephadm-2:18.2.1-341.g4e3182a6.el8.noarch 111/161 2024-01-24T13:28:03.762 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : cephadm-2:18.2.1-341.g4e3182a6.el8.noarch 111/161 2024-01-24T13:28:03.925 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-prometheus-alerts-2:18.2.1-341.g4e3182a6.el 112/161 2024-01-24T13:28:04.112 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-grafana-dashboards-2:18.2.1-341.g4e3182a6.e 113/161 2024-01-24T13:28:04.335 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-cephadm-2:18.2.1-341.g4e3182a6.el8.noar 114/161 2024-01-24T13:28:04.562 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.1-341.g4e3182a6.el8.noar 114/161 2024-01-24T13:28:04.875 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-dashboard-2:18.2.1-341.g4e3182a6.el8.no 115/161 2024-01-24T13:28:05.844 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.1-341.g4e3182a6.el8.no 115/161 2024-01-24T13:28:05.985 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.1-341.g4e31 116/161 2024-01-24T13:28:06.080 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.1-341.g4e31 116/161 2024-01-24T13:28:06.421 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-modules-core-2:18.2.1-341.g4e3182a6.el8 117/161 2024-01-24T13:28:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: rbd-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 124/161 2024-01-24T13:28:06.593 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-2:18.2.1-341.g4e3182a6.el8.x86_64 118/161 2024-01-24T13:28:06.630 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-2:18.2.1-341.g4e3182a6.el8.x86_64 118/161 2024-01-24T13:28:06.696 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-selinux-2:18.2.1-341.g4e3182a6.el8.x86_64 106/161 2024-01-24T13:28:06.696 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /sys 2024-01-24T13:28:06.696 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /proc 2024-01-24T13:28:06.696 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /dev 2024-01-24T13:28:06.696 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /run 2024-01-24T13:28:06.696 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /mnt 2024-01-24T13:28:06.696 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /var/tmp 2024-01-24T13:28:06.696 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /home 2024-01-24T13:28:06.696 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /tmp 2024-01-24T13:28:06.696 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /dev 2024-01-24T13:28:06.697 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:28:06.777 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-rook-2:18.2.1-341.g4e3182a6.el8.noarch 119/161 2024-01-24T13:28:06.789 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.1-341.g4e3182a6.el8.noarch 119/161 2024-01-24T13:28:06.994 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-test-2:18.2.1-341.g4e3182a6.el8.x86_64 125/161 2024-01-24T13:28:07.074 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-volume-2:18.2.1-341.g4e3182a6.el8.noarch 107/161 2024-01-24T13:28:07.356 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : postgresql-contrib-10.23-3.module+el8.9.0+20831+ 126/161 2024-01-24T13:28:07.562 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libtool-2.4.6-25.el8.x86_64 127/161 2024-01-24T13:28:08.037 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-2:18.2.1-341.g4e3182a6.el8.x86_64 120/161 2024-01-24T13:28:08.244 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 127/161 2024-01-24T13:28:08.367 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-radosgw-2:18.2.1-341.g4e3182a6.el8.x86_64 121/161 2024-01-24T13:28:08.403 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-radosgw-2:18.2.1-341.g4e3182a6.el8.x86_64 121/161 2024-01-24T13:28:08.525 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : openssl-devel-1:1.1.1k-12.el8_9.x86_64 128/161 2024-01-24T13:28:08.619 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-immutable-object-cache-2:18.2.1-341.g4e3182 122/161 2024-01-24T13:28:08.662 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.1-341.g4e3182 122/161 2024-01-24T13:28:08.811 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-volume-2:18.2.1-341.g4e3182a6.el8.noarch 107/161 2024-01-24T13:28:08.836 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : elfutils-libelf-devel-0.189-3.el8.x86_64 129/161 2024-01-24T13:28:08.869 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : cephfs-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 123/161 2024-01-24T13:28:09.006 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-osd-2:18.2.1-341.g4e3182a6.el8.x86_64 108/161 2024-01-24T13:28:09.043 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-fuse-2:18.2.1-341.g4e3182a6.el8.x86_64 130/161 2024-01-24T13:28:09.215 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: cephfs-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 123/161 2024-01-24T13:28:09.251 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-osd-2:18.2.1-341.g4e3182a6.el8.x86_64 108/161 2024-01-24T13:28:09.323 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : util-linux-user-2.32.1-43.el8.x86_64 131/161 2024-01-24T13:28:09.416 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mds-2:18.2.1-341.g4e3182a6.el8.x86_64 109/161 2024-01-24T13:28:09.435 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 124/161 2024-01-24T13:28:09.516 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 132/161 2024-01-24T13:28:09.696 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libcephfs-devel-2:18.2.1-341.g4e3182a6.el8.x86_6 133/161 2024-01-24T13:28:09.923 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mds-2:18.2.1-341.g4e3182a6.el8.x86_64 109/161 2024-01-24T13:28:10.075 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mon-2:18.2.1-341.g4e3182a6.el8.x86_64 110/161 2024-01-24T13:28:10.097 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mon-2:18.2.1-341.g4e3182a6.el8.x86_64 110/161 2024-01-24T13:28:11.934 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: postgresql-server-10.23-3.module+el8.9.0+20831+7 134/161 2024-01-24T13:28:12.090 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephadm-2:18.2.1-341.g4e3182a6.el8.noarch 111/161 2024-01-24T13:28:12.229 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : postgresql-server-10.23-3.module+el8.9.0+20831+7 134/161 2024-01-24T13:28:12.242 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephadm-2:18.2.1-341.g4e3182a6.el8.noarch 111/161 2024-01-24T13:28:12.281 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: postgresql-server-10.23-3.module+el8.9.0+20831+7 134/161 2024-01-24T13:28:12.373 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-prometheus-alerts-2:18.2.1-341.g4e3182a6.el 112/161 2024-01-24T13:28:12.573 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-grafana-dashboards-2:18.2.1-341.g4e3182a6.e 113/161 2024-01-24T13:28:12.687 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : elfutils-libs-0.189-3.el8.x86_64 135/161 2024-01-24T13:28:12.839 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-cephadm-2:18.2.1-341.g4e3182a6.el8.noar 114/161 2024-01-24T13:28:13.060 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : iproute-6.2.0-5.el8_9.x86_64 136/161 2024-01-24T13:28:13.083 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.1-341.g4e3182a6.el8.noar 114/161 2024-01-24T13:28:13.341 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : xfsdump-3.1.8-7.el8_9.x86_64 137/161 2024-01-24T13:28:13.447 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-dashboard-2:18.2.1-341.g4e3182a6.el8.no 115/161 2024-01-24T13:28:13.528 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : cephfs-top-2:18.2.1-341.g4e3182a6.el8.noarch 138/161 2024-01-24T13:28:13.736 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : rbd-fuse-2:18.2.1-341.g4e3182a6.el8.x86_64 139/161 2024-01-24T13:28:13.865 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: rbd-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 124/161 2024-01-24T13:28:14.035 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : rbd-nbd-2:18.2.1-341.g4e3182a6.el8.x86_64 140/161 2024-01-24T13:28:14.288 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : openssl-1:1.1.1k-12.el8_9.x86_64 141/161 2024-01-24T13:28:14.330 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-test-2:18.2.1-341.g4e3182a6.el8.x86_64 125/161 2024-01-24T13:28:14.486 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : NetworkManager-initscripts-updown-1:1.40.16-13.e 142/161 2024-01-24T13:28:14.495 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-13.e 142/161 2024-01-24T13:28:14.526 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.1-341.g4e3182a6.el8.no 115/161 2024-01-24T13:28:14.652 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : postgresql-contrib-10.23-3.module+el8.9.0+20831+ 126/161 2024-01-24T13:28:14.663 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 143/161 2024-01-24T13:28:14.706 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.1-341.g4e31 116/161 2024-01-24T13:28:14.814 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.1-341.g4e31 116/161 2024-01-24T13:28:14.850 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libtool-2.4.6-25.el8.x86_64 127/161 2024-01-24T13:28:14.902 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : flex-2.6.1-9.el8.x86_64 144/161 2024-01-24T13:28:14.978 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 144/161 2024-01-24T13:28:15.166 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-modules-core-2:18.2.1-341.g4e3182a6.el8 117/161 2024-01-24T13:28:15.203 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : bison-3.0.4-10.el8.x86_64 145/161 2024-01-24T13:28:15.216 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 145/161 2024-01-24T13:28:15.331 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-2:18.2.1-341.g4e3182a6.el8.x86_64 118/161 2024-01-24T13:28:15.376 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-2:18.2.1-341.g4e3182a6.el8.x86_64 118/161 2024-01-24T13:28:15.413 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : elfutils-libs-0.186-1.el8.x86_64 146/161 2024-01-24T13:28:15.539 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-rook-2:18.2.1-341.g4e3182a6.el8.noarch 119/161 2024-01-24T13:28:15.551 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 127/161 2024-01-24T13:28:15.555 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.1-341.g4e3182a6.el8.noarch 119/161 2024-01-24T13:28:15.587 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 147/161 2024-01-24T13:28:15.781 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 148/161 2024-01-24T13:28:15.781 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 149/161 2024-01-24T13:28:15.872 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : openssl-devel-1:1.1.1k-12.el8_9.x86_64 128/161 2024-01-24T13:28:16.089 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 149/161 2024-01-24T13:28:16.089 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 150/161 2024-01-24T13:28:16.149 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : elfutils-libelf-devel-0.189-3.el8.x86_64 129/161 2024-01-24T13:28:16.357 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-fuse-2:18.2.1-341.g4e3182a6.el8.x86_64 130/161 2024-01-24T13:28:16.373 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 150/161 2024-01-24T13:28:16.373 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 151/161 2024-01-24T13:28:16.595 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : util-linux-user-2.32.1-43.el8.x86_64 131/161 2024-01-24T13:28:16.650 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 151/161 2024-01-24T13:28:16.792 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 132/161 2024-01-24T13:28:16.904 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 152/161 2024-01-24T13:28:16.965 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-2:18.2.1-341.g4e3182a6.el8.x86_64 120/161 2024-01-24T13:28:16.976 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephfs-devel-2:18.2.1-341.g4e3182a6.el8.x86_6 133/161 2024-01-24T13:28:17.087 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : util-linux-user-2.32.1-35.el8.x86_64 153/161 2024-01-24T13:28:17.222 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-radosgw-2:18.2.1-341.g4e3182a6.el8.x86_64 121/161 2024-01-24T13:28:17.264 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-radosgw-2:18.2.1-341.g4e3182a6.el8.x86_64 121/161 2024-01-24T13:28:17.354 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : util-linux-2.32.1-35.el8.x86_64 154/161 2024-01-24T13:28:17.354 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : libfdisk-2.32.1-35.el8.x86_64 155/161 2024-01-24T13:28:17.432 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-immutable-object-cache-2:18.2.1-341.g4e3182 122/161 2024-01-24T13:28:17.482 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.1-341.g4e3182 122/161 2024-01-24T13:28:17.605 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libfdisk-2.32.1-35.el8.x86_64 155/161 2024-01-24T13:28:17.605 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : libmount-2.32.1-35.el8.x86_64 156/161 2024-01-24T13:28:17.656 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephfs-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 123/161 2024-01-24T13:28:17.821 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libmount-2.32.1-35.el8.x86_64 156/161 2024-01-24T13:28:17.822 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : libblkid-2.32.1-35.el8.x86_64 157/161 2024-01-24T13:28:18.044 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephfs-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 123/161 2024-01-24T13:28:18.072 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libblkid-2.32.1-35.el8.x86_64 157/161 2024-01-24T13:28:18.072 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : libuuid-2.32.1-35.el8.x86_64 158/161 2024-01-24T13:28:18.223 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 124/161 2024-01-24T13:28:18.288 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libuuid-2.32.1-35.el8.x86_64 158/161 2024-01-24T13:28:18.288 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : libsmartcols-2.32.1-35.el8.x86_64 159/161 2024-01-24T13:28:18.546 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libsmartcols-2.32.1-35.el8.x86_64 159/161 2024-01-24T13:28:18.697 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : elfutils-libelf-0.186-1.el8.x86_64 160/161 2024-01-24T13:28:18.697 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 161/161 2024-01-24T13:28:19.211 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: postgresql-server-10.23-3.module+el8.9.0+20831+7 134/161 2024-01-24T13:28:19.475 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : postgresql-server-10.23-3.module+el8.9.0+20831+7 134/161 2024-01-24T13:28:19.515 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: postgresql-server-10.23-3.module+el8.9.0+20831+7 134/161 2024-01-24T13:28:19.625 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 161/161 2024-01-24T13:28:19.625 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-2:18.2.1-341.g4e3182a6.el8.x86_64 1/161 2024-01-24T13:28:19.625 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-base-2:18.2.1-341.g4e3182a6.el8.x86_64 2/161 2024-01-24T13:28:19.626 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-common-2:18.2.1-341.g4e3182a6.el8.x86_64 3/161 2024-01-24T13:28:19.626 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-fuse-2:18.2.1-341.g4e3182a6.el8.x86_64 4/161 2024-01-24T13:28:19.626 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-immutable-object-cache-2:18.2.1-341.g4e3182 5/161 2024-01-24T13:28:19.626 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mds-2:18.2.1-341.g4e3182a6.el8.x86_64 6/161 2024-01-24T13:28:19.626 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-2:18.2.1-341.g4e3182a6.el8.x86_64 7/161 2024-01-24T13:28:19.626 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mon-2:18.2.1-341.g4e3182a6.el8.x86_64 8/161 2024-01-24T13:28:19.626 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-osd-2:18.2.1-341.g4e3182a6.el8.x86_64 9/161 2024-01-24T13:28:19.626 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-radosgw-2:18.2.1-341.g4e3182a6.el8.x86_64 10/161 2024-01-24T13:28:19.626 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-selinux-2:18.2.1-341.g4e3182a6.el8.x86_64 11/161 2024-01-24T13:28:19.626 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-test-2:18.2.1-341.g4e3182a6.el8.x86_64 12/161 2024-01-24T13:28:19.626 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : cephfs-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 13/161 2024-01-24T13:28:19.626 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libcephfs-devel-2:18.2.1-341.g4e3182a6.el8.x86_6 14/161 2024-01-24T13:28:19.626 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libcephfs2-2:18.2.1-341.g4e3182a6.el8.x86_64 15/161 2024-01-24T13:28:19.627 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libcephsqlite-2:18.2.1-341.g4e3182a6.el8.x86_64 16/161 2024-01-24T13:28:19.627 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librados-devel-2:18.2.1-341.g4e3182a6.el8.x86_64 17/161 2024-01-24T13:28:19.627 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libradosstriper1-2:18.2.1-341.g4e3182a6.el8.x86_ 18/161 2024-01-24T13:28:19.627 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librgw2-2:18.2.1-341.g4e3182a6.el8.x86_64 19/161 2024-01-24T13:28:19.627 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-ceph-argparse-2:18.2.1-341.g4e3182a6.el8 20/161 2024-01-24T13:28:19.627 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-ceph-common-2:18.2.1-341.g4e3182a6.el8.x 21/161 2024-01-24T13:28:19.627 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cephfs-2:18.2.1-341.g4e3182a6.el8.x86_64 22/161 2024-01-24T13:28:19.627 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rados-2:18.2.1-341.g4e3182a6.el8.x86_64 23/161 2024-01-24T13:28:19.627 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rbd-2:18.2.1-341.g4e3182a6.el8.x86_64 24/161 2024-01-24T13:28:19.627 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rgw-2:18.2.1-341.g4e3182a6.el8.x86_64 25/161 2024-01-24T13:28:19.627 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : rbd-fuse-2:18.2.1-341.g4e3182a6.el8.x86_64 26/161 2024-01-24T13:28:19.627 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : rbd-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 27/161 2024-01-24T13:28:19.627 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : rbd-nbd-2:18.2.1-341.g4e3182a6.el8.x86_64 28/161 2024-01-24T13:28:19.627 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-grafana-dashboards-2:18.2.1-341.g4e3182a6.e 29/161 2024-01-24T13:28:19.627 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-cephadm-2:18.2.1-341.g4e3182a6.el8.noar 30/161 2024-01-24T13:28:19.639 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-dashboard-2:18.2.1-341.g4e3182a6.el8.no 31/161 2024-01-24T13:28:19.639 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.1-341.g4e31 32/161 2024-01-24T13:28:19.640 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-modules-core-2:18.2.1-341.g4e3182a6.el8 33/161 2024-01-24T13:28:19.640 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-rook-2:18.2.1-341.g4e3182a6.el8.noarch 34/161 2024-01-24T13:28:19.640 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-prometheus-alerts-2:18.2.1-341.g4e3182a6.el 35/161 2024-01-24T13:28:19.640 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-volume-2:18.2.1-341.g4e3182a6.el8.noarch 36/161 2024-01-24T13:28:19.640 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : cephadm-2:18.2.1-341.g4e3182a6.el8.noarch 37/161 2024-01-24T13:28:19.640 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : cephfs-top-2:18.2.1-341.g4e3182a6.el8.noarch 38/161 2024-01-24T13:28:19.640 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/161 2024-01-24T13:28:19.640 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/161 2024-01-24T13:28:19.640 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/161 2024-01-24T13:28:19.640 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/161 2024-01-24T13:28:19.640 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/161 2024-01-24T13:28:19.640 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/161 2024-01-24T13:28:19.641 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/161 2024-01-24T13:28:19.641 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/161 2024-01-24T13:28:19.641 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/161 2024-01-24T13:28:19.641 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/161 2024-01-24T13:28:19.641 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/161 2024-01-24T13:28:19.641 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/161 2024-01-24T13:28:19.641 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/161 2024-01-24T13:28:19.641 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/161 2024-01-24T13:28:19.641 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/161 2024-01-24T13:28:19.641 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/161 2024-01-24T13:28:19.641 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/161 2024-01-24T13:28:19.641 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/161 2024-01-24T13:28:19.641 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/161 2024-01-24T13:28:19.641 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/161 2024-01-24T13:28:19.641 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/161 2024-01-24T13:28:19.641 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/161 2024-01-24T13:28:19.642 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/161 2024-01-24T13:28:19.642 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/161 2024-01-24T13:28:19.642 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/161 2024-01-24T13:28:19.642 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/161 2024-01-24T13:28:19.642 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/161 2024-01-24T13:28:19.642 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/161 2024-01-24T13:28:19.642 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/161 2024-01-24T13:28:19.642 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/161 2024-01-24T13:28:19.642 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/161 2024-01-24T13:28:19.642 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/161 2024-01-24T13:28:19.642 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/161 2024-01-24T13:28:19.642 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 72/161 2024-01-24T13:28:19.642 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 73/161 2024-01-24T13:28:19.642 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 74/161 2024-01-24T13:28:19.642 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 75/161 2024-01-24T13:28:19.643 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 76/161 2024-01-24T13:28:19.643 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 77/161 2024-01-24T13:28:19.643 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/161 2024-01-24T13:28:19.643 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : flex-2.6.1-9.el8.x86_64 79/161 2024-01-24T13:28:19.643 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 80/161 2024-01-24T13:28:19.643 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 81/161 2024-01-24T13:28:19.643 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 82/161 2024-01-24T13:28:19.643 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 83/161 2024-01-24T13:28:19.643 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 84/161 2024-01-24T13:28:19.643 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 85/161 2024-01-24T13:28:19.643 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 86/161 2024-01-24T13:28:19.643 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 87/161 2024-01-24T13:28:19.643 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 88/161 2024-01-24T13:28:19.643 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 89/161 2024-01-24T13:28:19.643 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 90/161 2024-01-24T13:28:19.643 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 91/161 2024-01-24T13:28:19.644 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 92/161 2024-01-24T13:28:19.644 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 93/161 2024-01-24T13:28:19.644 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 94/161 2024-01-24T13:28:19.644 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : automake-1.16.1-8.el8.noarch 95/161 2024-01-24T13:28:19.644 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libpq-13.11-1.el8.x86_64 96/161 2024-01-24T13:28:19.644 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : jq-1.6-7.el8.x86_64 97/161 2024-01-24T13:28:19.644 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 98/161 2024-01-24T13:28:19.644 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : postgresql-10.23-3.module+el8.9.0+20831+78489779 99/161 2024-01-24T13:28:19.644 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : postgresql-contrib-10.23-3.module+el8.9.0+20831+ 100/161 2024-01-24T13:28:19.644 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : postgresql-server-10.23-3.module+el8.9.0+20831+7 101/161 2024-01-24T13:28:19.644 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 102/161 2024-01-24T13:28:19.644 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 103/161 2024-01-24T13:28:19.644 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 104/161 2024-01-24T13:28:19.645 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 105/161 2024-01-24T13:28:19.645 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 106/161 2024-01-24T13:28:19.645 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 107/161 2024-01-24T13:28:19.645 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 108/161 2024-01-24T13:28:19.645 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 109/161 2024-01-24T13:28:19.645 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 110/161 2024-01-24T13:28:19.645 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/161 2024-01-24T13:28:19.645 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 112/161 2024-01-24T13:28:19.645 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 113/161 2024-01-24T13:28:19.645 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 114/161 2024-01-24T13:28:19.645 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 115/161 2024-01-24T13:28:19.645 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 116/161 2024-01-24T13:28:19.645 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 117/161 2024-01-24T13:28:19.645 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 118/161 2024-01-24T13:28:19.645 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 119/161 2024-01-24T13:28:19.646 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 120/161 2024-01-24T13:28:19.646 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libuuid-devel-2.32.1-43.el8.x86_64 121/161 2024-01-24T13:28:19.646 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 122/161 2024-01-24T13:28:19.646 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 123/161 2024-01-24T13:28:19.646 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 124/161 2024-01-24T13:28:19.646 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : elfutils-libelf-devel-0.189-3.el8.x86_64 125/161 2024-01-24T13:28:19.646 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 126/161 2024-01-24T13:28:19.646 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : krb5-devel-1.18.2-26.el8_9.x86_64 127/161 2024-01-24T13:28:19.646 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8_9.x86_64 128/161 2024-01-24T13:28:19.646 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-13.e 129/161 2024-01-24T13:28:19.646 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librados2-2:18.2.1-341.g4e3182a6.el8.x86_64 130/161 2024-01-24T13:28:19.646 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 131/161 2024-01-24T13:28:19.646 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librbd1-2:18.2.1-341.g4e3182a6.el8.x86_64 132/161 2024-01-24T13:28:19.646 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 133/161 2024-01-24T13:28:19.647 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : elfutils-libelf-0.189-3.el8.x86_64 134/161 2024-01-24T13:28:19.647 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : elfutils-libelf-0.186-1.el8.x86_64 135/161 2024-01-24T13:28:19.647 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : elfutils-libs-0.189-3.el8.x86_64 136/161 2024-01-24T13:28:19.647 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : elfutils-libs-0.186-1.el8.x86_64 137/161 2024-01-24T13:28:19.647 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libmount-2.32.1-43.el8.x86_64 138/161 2024-01-24T13:28:19.647 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libmount-2.32.1-35.el8.x86_64 139/161 2024-01-24T13:28:19.647 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libsmartcols-2.32.1-43.el8.x86_64 140/161 2024-01-24T13:28:19.647 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libsmartcols-2.32.1-35.el8.x86_64 141/161 2024-01-24T13:28:19.647 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libuuid-2.32.1-43.el8.x86_64 142/161 2024-01-24T13:28:19.647 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libuuid-2.32.1-35.el8.x86_64 143/161 2024-01-24T13:28:19.647 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : util-linux-2.32.1-43.el8.x86_64 144/161 2024-01-24T13:28:19.647 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : util-linux-2.32.1-35.el8.x86_64 145/161 2024-01-24T13:28:19.647 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : util-linux-user-2.32.1-43.el8.x86_64 146/161 2024-01-24T13:28:19.647 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : util-linux-user-2.32.1-35.el8.x86_64 147/161 2024-01-24T13:28:19.647 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : iproute-6.2.0-5.el8_9.x86_64 148/161 2024-01-24T13:28:19.648 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 149/161 2024-01-24T13:28:19.648 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libblkid-2.32.1-43.el8.x86_64 150/161 2024-01-24T13:28:19.648 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libblkid-2.32.1-35.el8.x86_64 151/161 2024-01-24T13:28:19.648 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libfdisk-2.32.1-43.el8.x86_64 152/161 2024-01-24T13:28:19.648 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libfdisk-2.32.1-35.el8.x86_64 153/161 2024-01-24T13:28:19.648 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 154/161 2024-01-24T13:28:19.648 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 155/161 2024-01-24T13:28:19.648 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openssl-1:1.1.1k-12.el8_9.x86_64 156/161 2024-01-24T13:28:19.648 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 157/161 2024-01-24T13:28:19.648 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8_9.x86_64 158/161 2024-01-24T13:28:19.648 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 159/161 2024-01-24T13:28:19.648 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : xfsdump-3.1.8-7.el8_9.x86_64 160/161 2024-01-24T13:28:19.969 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : elfutils-libs-0.189-3.el8.x86_64 135/161 2024-01-24T13:28:20.349 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : iproute-6.2.0-5.el8_9.x86_64 136/161 2024-01-24T13:28:20.645 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : xfsdump-3.1.8-7.el8_9.x86_64 137/161 2024-01-24T13:28:20.850 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : cephfs-top-2:18.2.1-341.g4e3182a6.el8.noarch 138/161 2024-01-24T13:28:21.049 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-fuse-2:18.2.1-341.g4e3182a6.el8.x86_64 139/161 2024-01-24T13:28:21.360 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-nbd-2:18.2.1-341.g4e3182a6.el8.x86_64 140/161 2024-01-24T13:28:21.651 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : openssl-1:1.1.1k-12.el8_9.x86_64 141/161 2024-01-24T13:28:21.833 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : NetworkManager-initscripts-updown-1:1.40.16-13.e 142/161 2024-01-24T13:28:21.842 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-13.e 142/161 2024-01-24T13:28:22.004 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 143/161 2024-01-24T13:28:22.257 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : flex-2.6.1-9.el8.x86_64 144/161 2024-01-24T13:28:22.338 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 144/161 2024-01-24T13:28:22.574 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : bison-3.0.4-10.el8.x86_64 145/161 2024-01-24T13:28:22.588 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 145/161 2024-01-24T13:28:22.795 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : elfutils-libs-0.186-1.el8.x86_64 146/161 2024-01-24T13:28:23.009 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 147/161 2024-01-24T13:28:23.131 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 124/161 2024-01-24T13:28:23.178 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 148/161 2024-01-24T13:28:23.178 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 149/161 2024-01-24T13:28:23.478 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 149/161 2024-01-24T13:28:23.478 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 150/161 2024-01-24T13:28:23.624 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-test-2:18.2.1-341.g4e3182a6.el8.x86_64 125/161 2024-01-24T13:28:23.728 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 150/161 2024-01-24T13:28:23.728 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 151/161 2024-01-24T13:28:23.916 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : postgresql-contrib-10.23-3.module+el8.9.0+20831+ 126/161 2024-01-24T13:28:24.022 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 151/161 2024-01-24T13:28:24.138 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libtool-2.4.6-25.el8.x86_64 127/161 2024-01-24T13:28:24.251 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 152/161 2024-01-24T13:28:24.443 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : util-linux-user-2.32.1-35.el8.x86_64 153/161 2024-01-24T13:28:24.726 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : util-linux-2.32.1-35.el8.x86_64 154/161 2024-01-24T13:28:24.726 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : libfdisk-2.32.1-35.el8.x86_64 155/161 2024-01-24T13:28:24.958 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 127/161 2024-01-24T13:28:24.993 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libfdisk-2.32.1-35.el8.x86_64 155/161 2024-01-24T13:28:24.993 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : libmount-2.32.1-35.el8.x86_64 156/161 2024-01-24T13:28:25.252 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libmount-2.32.1-35.el8.x86_64 156/161 2024-01-24T13:28:25.252 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : libblkid-2.32.1-35.el8.x86_64 157/161 2024-01-24T13:28:25.294 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openssl-devel-1:1.1.1k-12.el8_9.x86_64 128/161 2024-01-24T13:28:25.518 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libblkid-2.32.1-35.el8.x86_64 157/161 2024-01-24T13:28:25.519 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : libuuid-2.32.1-35.el8.x86_64 158/161 2024-01-24T13:28:25.588 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : elfutils-libelf-devel-0.189-3.el8.x86_64 129/161 2024-01-24T13:28:25.726 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libuuid-2.32.1-35.el8.x86_64 158/161 2024-01-24T13:28:25.727 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : libsmartcols-2.32.1-35.el8.x86_64 159/161 2024-01-24T13:28:25.771 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-fuse-2:18.2.1-341.g4e3182a6.el8.x86_64 130/161 2024-01-24T13:28:25.960 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libsmartcols-2.32.1-35.el8.x86_64 159/161 2024-01-24T13:28:26.044 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : util-linux-user-2.32.1-43.el8.x86_64 131/161 2024-01-24T13:28:26.152 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : elfutils-libelf-0.186-1.el8.x86_64 160/161 2024-01-24T13:28:26.152 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 161/161 2024-01-24T13:28:26.235 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 132/161 2024-01-24T13:28:26.431 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs-devel-2:18.2.1-341.g4e3182a6.el8.x86_6 133/161 2024-01-24T13:28:27.023 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 161/161 2024-01-24T13:28:27.092 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-2:18.2.1-341.g4e3182a6.el8.x86_64 1/161 2024-01-24T13:28:27.092 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-base-2:18.2.1-341.g4e3182a6.el8.x86_64 2/161 2024-01-24T13:28:27.095 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-common-2:18.2.1-341.g4e3182a6.el8.x86_64 3/161 2024-01-24T13:28:27.095 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-fuse-2:18.2.1-341.g4e3182a6.el8.x86_64 4/161 2024-01-24T13:28:27.095 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-immutable-object-cache-2:18.2.1-341.g4e3182 5/161 2024-01-24T13:28:27.095 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mds-2:18.2.1-341.g4e3182a6.el8.x86_64 6/161 2024-01-24T13:28:27.095 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-2:18.2.1-341.g4e3182a6.el8.x86_64 7/161 2024-01-24T13:28:27.095 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mon-2:18.2.1-341.g4e3182a6.el8.x86_64 8/161 2024-01-24T13:28:27.095 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-osd-2:18.2.1-341.g4e3182a6.el8.x86_64 9/161 2024-01-24T13:28:27.095 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-radosgw-2:18.2.1-341.g4e3182a6.el8.x86_64 10/161 2024-01-24T13:28:27.095 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-selinux-2:18.2.1-341.g4e3182a6.el8.x86_64 11/161 2024-01-24T13:28:27.095 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-test-2:18.2.1-341.g4e3182a6.el8.x86_64 12/161 2024-01-24T13:28:27.096 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : cephfs-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 13/161 2024-01-24T13:28:27.096 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephfs-devel-2:18.2.1-341.g4e3182a6.el8.x86_6 14/161 2024-01-24T13:28:27.096 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephfs2-2:18.2.1-341.g4e3182a6.el8.x86_64 15/161 2024-01-24T13:28:27.096 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephsqlite-2:18.2.1-341.g4e3182a6.el8.x86_64 16/161 2024-01-24T13:28:27.096 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados-devel-2:18.2.1-341.g4e3182a6.el8.x86_64 17/161 2024-01-24T13:28:27.096 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libradosstriper1-2:18.2.1-341.g4e3182a6.el8.x86_ 18/161 2024-01-24T13:28:27.096 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librgw2-2:18.2.1-341.g4e3182a6.el8.x86_64 19/161 2024-01-24T13:28:27.096 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-ceph-argparse-2:18.2.1-341.g4e3182a6.el8 20/161 2024-01-24T13:28:27.096 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-ceph-common-2:18.2.1-341.g4e3182a6.el8.x 21/161 2024-01-24T13:28:27.096 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cephfs-2:18.2.1-341.g4e3182a6.el8.x86_64 22/161 2024-01-24T13:28:27.096 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rados-2:18.2.1-341.g4e3182a6.el8.x86_64 23/161 2024-01-24T13:28:27.096 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rbd-2:18.2.1-341.g4e3182a6.el8.x86_64 24/161 2024-01-24T13:28:27.097 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rgw-2:18.2.1-341.g4e3182a6.el8.x86_64 25/161 2024-01-24T13:28:27.097 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-fuse-2:18.2.1-341.g4e3182a6.el8.x86_64 26/161 2024-01-24T13:28:27.097 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 27/161 2024-01-24T13:28:27.097 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-nbd-2:18.2.1-341.g4e3182a6.el8.x86_64 28/161 2024-01-24T13:28:27.097 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-grafana-dashboards-2:18.2.1-341.g4e3182a6.e 29/161 2024-01-24T13:28:27.109 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-cephadm-2:18.2.1-341.g4e3182a6.el8.noar 30/161 2024-01-24T13:28:27.109 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-dashboard-2:18.2.1-341.g4e3182a6.el8.no 31/161 2024-01-24T13:28:27.109 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.1-341.g4e31 32/161 2024-01-24T13:28:27.110 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-modules-core-2:18.2.1-341.g4e3182a6.el8 33/161 2024-01-24T13:28:27.110 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-rook-2:18.2.1-341.g4e3182a6.el8.noarch 34/161 2024-01-24T13:28:27.110 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-prometheus-alerts-2:18.2.1-341.g4e3182a6.el 35/161 2024-01-24T13:28:27.110 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-volume-2:18.2.1-341.g4e3182a6.el8.noarch 36/161 2024-01-24T13:28:27.110 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : cephadm-2:18.2.1-341.g4e3182a6.el8.noarch 37/161 2024-01-24T13:28:27.110 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : cephfs-top-2:18.2.1-341.g4e3182a6.el8.noarch 38/161 2024-01-24T13:28:27.110 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/161 2024-01-24T13:28:27.110 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/161 2024-01-24T13:28:27.110 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/161 2024-01-24T13:28:27.110 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/161 2024-01-24T13:28:27.110 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/161 2024-01-24T13:28:27.110 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/161 2024-01-24T13:28:27.110 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/161 2024-01-24T13:28:27.111 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/161 2024-01-24T13:28:27.111 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/161 2024-01-24T13:28:27.111 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/161 2024-01-24T13:28:27.111 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/161 2024-01-24T13:28:27.111 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/161 2024-01-24T13:28:27.111 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/161 2024-01-24T13:28:27.111 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/161 2024-01-24T13:28:27.111 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/161 2024-01-24T13:28:27.111 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/161 2024-01-24T13:28:27.111 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/161 2024-01-24T13:28:27.111 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/161 2024-01-24T13:28:27.111 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/161 2024-01-24T13:28:27.111 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/161 2024-01-24T13:28:27.111 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/161 2024-01-24T13:28:27.112 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/161 2024-01-24T13:28:27.112 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/161 2024-01-24T13:28:27.112 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/161 2024-01-24T13:28:27.112 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/161 2024-01-24T13:28:27.112 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/161 2024-01-24T13:28:27.112 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/161 2024-01-24T13:28:27.112 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/161 2024-01-24T13:28:27.112 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/161 2024-01-24T13:28:27.112 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/161 2024-01-24T13:28:27.112 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/161 2024-01-24T13:28:27.112 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/161 2024-01-24T13:28:27.112 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/161 2024-01-24T13:28:27.113 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 72/161 2024-01-24T13:28:27.113 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 73/161 2024-01-24T13:28:27.113 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 74/161 2024-01-24T13:28:27.113 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 75/161 2024-01-24T13:28:27.113 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 76/161 2024-01-24T13:28:27.113 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 77/161 2024-01-24T13:28:27.113 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/161 2024-01-24T13:28:27.113 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : flex-2.6.1-9.el8.x86_64 79/161 2024-01-24T13:28:27.113 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 80/161 2024-01-24T13:28:27.113 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 81/161 2024-01-24T13:28:27.113 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 82/161 2024-01-24T13:28:27.113 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 83/161 2024-01-24T13:28:27.113 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 84/161 2024-01-24T13:28:27.114 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 85/161 2024-01-24T13:28:27.114 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 86/161 2024-01-24T13:28:27.114 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 87/161 2024-01-24T13:28:27.114 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 88/161 2024-01-24T13:28:27.114 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 89/161 2024-01-24T13:28:27.114 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 90/161 2024-01-24T13:28:27.114 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 91/161 2024-01-24T13:28:27.114 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 92/161 2024-01-24T13:28:27.114 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 93/161 2024-01-24T13:28:27.114 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 94/161 2024-01-24T13:28:27.114 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : automake-1.16.1-8.el8.noarch 95/161 2024-01-24T13:28:27.114 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libpq-13.11-1.el8.x86_64 96/161 2024-01-24T13:28:27.114 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : jq-1.6-7.el8.x86_64 97/161 2024-01-24T13:28:27.114 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 98/161 2024-01-24T13:28:27.115 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : postgresql-10.23-3.module+el8.9.0+20831+78489779 99/161 2024-01-24T13:28:27.115 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : postgresql-contrib-10.23-3.module+el8.9.0+20831+ 100/161 2024-01-24T13:28:27.115 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : postgresql-server-10.23-3.module+el8.9.0+20831+7 101/161 2024-01-24T13:28:27.115 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 102/161 2024-01-24T13:28:27.115 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 103/161 2024-01-24T13:28:27.115 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 104/161 2024-01-24T13:28:27.115 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 105/161 2024-01-24T13:28:27.115 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 106/161 2024-01-24T13:28:27.115 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 107/161 2024-01-24T13:28:27.115 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 108/161 2024-01-24T13:28:27.115 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 109/161 2024-01-24T13:28:27.115 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 110/161 2024-01-24T13:28:27.115 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/161 2024-01-24T13:28:27.116 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 112/161 2024-01-24T13:28:27.116 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 113/161 2024-01-24T13:28:27.116 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 114/161 2024-01-24T13:28:27.116 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 115/161 2024-01-24T13:28:27.116 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 116/161 2024-01-24T13:28:27.116 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 117/161 2024-01-24T13:28:27.116 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 118/161 2024-01-24T13:28:27.116 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 119/161 2024-01-24T13:28:27.116 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 120/161 2024-01-24T13:28:27.116 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libuuid-devel-2.32.1-43.el8.x86_64 121/161 2024-01-24T13:28:27.116 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 122/161 2024-01-24T13:28:27.116 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 123/161 2024-01-24T13:28:27.116 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 124/161 2024-01-24T13:28:27.116 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : elfutils-libelf-devel-0.189-3.el8.x86_64 125/161 2024-01-24T13:28:27.116 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 126/161 2024-01-24T13:28:27.117 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : krb5-devel-1.18.2-26.el8_9.x86_64 127/161 2024-01-24T13:28:27.117 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8_9.x86_64 128/161 2024-01-24T13:28:27.117 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-13.e 129/161 2024-01-24T13:28:27.117 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados2-2:18.2.1-341.g4e3182a6.el8.x86_64 130/161 2024-01-24T13:28:27.117 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 131/161 2024-01-24T13:28:27.117 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librbd1-2:18.2.1-341.g4e3182a6.el8.x86_64 132/161 2024-01-24T13:28:27.117 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 133/161 2024-01-24T13:28:27.117 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : elfutils-libelf-0.189-3.el8.x86_64 134/161 2024-01-24T13:28:27.117 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : elfutils-libelf-0.186-1.el8.x86_64 135/161 2024-01-24T13:28:27.117 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : elfutils-libs-0.189-3.el8.x86_64 136/161 2024-01-24T13:28:27.117 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : elfutils-libs-0.186-1.el8.x86_64 137/161 2024-01-24T13:28:27.117 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libmount-2.32.1-43.el8.x86_64 138/161 2024-01-24T13:28:27.117 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libmount-2.32.1-35.el8.x86_64 139/161 2024-01-24T13:28:27.117 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libsmartcols-2.32.1-43.el8.x86_64 140/161 2024-01-24T13:28:27.117 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libsmartcols-2.32.1-35.el8.x86_64 141/161 2024-01-24T13:28:27.118 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libuuid-2.32.1-43.el8.x86_64 142/161 2024-01-24T13:28:27.118 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libuuid-2.32.1-35.el8.x86_64 143/161 2024-01-24T13:28:27.118 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : util-linux-2.32.1-43.el8.x86_64 144/161 2024-01-24T13:28:27.118 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : util-linux-2.32.1-35.el8.x86_64 145/161 2024-01-24T13:28:27.118 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : util-linux-user-2.32.1-43.el8.x86_64 146/161 2024-01-24T13:28:27.118 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : util-linux-user-2.32.1-35.el8.x86_64 147/161 2024-01-24T13:28:27.118 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : iproute-6.2.0-5.el8_9.x86_64 148/161 2024-01-24T13:28:27.118 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 149/161 2024-01-24T13:28:27.118 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libblkid-2.32.1-43.el8.x86_64 150/161 2024-01-24T13:28:27.118 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libblkid-2.32.1-35.el8.x86_64 151/161 2024-01-24T13:28:27.118 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libfdisk-2.32.1-43.el8.x86_64 152/161 2024-01-24T13:28:27.118 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libfdisk-2.32.1-35.el8.x86_64 153/161 2024-01-24T13:28:27.118 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 154/161 2024-01-24T13:28:27.118 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 155/161 2024-01-24T13:28:27.119 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openssl-1:1.1.1k-12.el8_9.x86_64 156/161 2024-01-24T13:28:27.119 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 157/161 2024-01-24T13:28:27.119 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8_9.x86_64 158/161 2024-01-24T13:28:27.119 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 159/161 2024-01-24T13:28:27.119 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : xfsdump-3.1.8-7.el8_9.x86_64 160/161 2024-01-24T13:28:28.937 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 161/161 2024-01-24T13:28:28.937 INFO:teuthology.orchestra.run.smithi187.stdout:Installed products updated. 2024-01-24T13:28:29.023 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: postgresql-server-10.23-3.module+el8.9.0+20831+7 134/161 2024-01-24T13:28:29.330 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : postgresql-server-10.23-3.module+el8.9.0+20831+7 134/161 2024-01-24T13:28:29.372 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: postgresql-server-10.23-3.module+el8.9.0+20831+7 134/161 2024-01-24T13:28:29.523 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-01-24T13:28:29.523 INFO:teuthology.orchestra.run.smithi187.stdout:Upgraded: 2024-01-24T13:28:29.523 INFO:teuthology.orchestra.run.smithi187.stdout: elfutils-libelf-0.189-3.el8.x86_64 2024-01-24T13:28:29.523 INFO:teuthology.orchestra.run.smithi187.stdout: elfutils-libs-0.189-3.el8.x86_64 2024-01-24T13:28:29.523 INFO:teuthology.orchestra.run.smithi187.stdout: iproute-6.2.0-5.el8_9.x86_64 2024-01-24T13:28:29.523 INFO:teuthology.orchestra.run.smithi187.stdout: libblkid-2.32.1-43.el8.x86_64 2024-01-24T13:28:29.523 INFO:teuthology.orchestra.run.smithi187.stdout: libfdisk-2.32.1-43.el8.x86_64 2024-01-24T13:28:29.523 INFO:teuthology.orchestra.run.smithi187.stdout: libmount-2.32.1-43.el8.x86_64 2024-01-24T13:28:29.523 INFO:teuthology.orchestra.run.smithi187.stdout: librados2-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:29.524 INFO:teuthology.orchestra.run.smithi187.stdout: librbd1-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:29.524 INFO:teuthology.orchestra.run.smithi187.stdout: libsmartcols-2.32.1-43.el8.x86_64 2024-01-24T13:28:29.524 INFO:teuthology.orchestra.run.smithi187.stdout: libuuid-2.32.1-43.el8.x86_64 2024-01-24T13:28:29.524 INFO:teuthology.orchestra.run.smithi187.stdout: openssl-1:1.1.1k-12.el8_9.x86_64 2024-01-24T13:28:29.524 INFO:teuthology.orchestra.run.smithi187.stdout: openssl-libs-1:1.1.1k-12.el8_9.x86_64 2024-01-24T13:28:29.524 INFO:teuthology.orchestra.run.smithi187.stdout: util-linux-2.32.1-43.el8.x86_64 2024-01-24T13:28:29.524 INFO:teuthology.orchestra.run.smithi187.stdout: util-linux-user-2.32.1-43.el8.x86_64 2024-01-24T13:28:29.524 INFO:teuthology.orchestra.run.smithi187.stdout: xfsdump-3.1.8-7.el8_9.x86_64 2024-01-24T13:28:29.524 INFO:teuthology.orchestra.run.smithi187.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-01-24T13:28:29.524 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-01-24T13:28:29.524 INFO:teuthology.orchestra.run.smithi187.stdout: NetworkManager-initscripts-updown-1:1.40.16-13.el8_9.noarch 2024-01-24T13:28:29.524 INFO:teuthology.orchestra.run.smithi187.stdout: automake-1.16.1-8.el8.noarch 2024-01-24T13:28:29.525 INFO:teuthology.orchestra.run.smithi187.stdout: bison-3.0.4-10.el8.x86_64 2024-01-24T13:28:29.525 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:29.525 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-base-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:29.525 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-common-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:29.525 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-fuse-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:29.525 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-grafana-dashboards-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T13:28:29.525 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-immutable-object-cache-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:29.525 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mds-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:29.525 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:29.525 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-cephadm-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T13:28:29.525 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-dashboard-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T13:28:29.525 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-diskprediction-local-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T13:28:29.525 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-modules-core-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T13:28:29.526 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-rook-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T13:28:29.526 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mon-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:29.526 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-osd-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:29.526 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-prometheus-alerts-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T13:28:29.526 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-radosgw-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:29.526 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-selinux-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:29.526 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-test-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:29.526 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-volume-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T13:28:29.526 INFO:teuthology.orchestra.run.smithi187.stdout: cephadm-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T13:28:29.526 INFO:teuthology.orchestra.run.smithi187.stdout: cephfs-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:29.526 INFO:teuthology.orchestra.run.smithi187.stdout: cephfs-top-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T13:28:29.526 INFO:teuthology.orchestra.run.smithi187.stdout: elfutils-libelf-devel-0.189-3.el8.x86_64 2024-01-24T13:28:29.526 INFO:teuthology.orchestra.run.smithi187.stdout: flex-2.6.1-9.el8.x86_64 2024-01-24T13:28:29.527 INFO:teuthology.orchestra.run.smithi187.stdout: fuse-2.9.7-17.el8.x86_64 2024-01-24T13:28:29.527 INFO:teuthology.orchestra.run.smithi187.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-24T13:28:29.527 INFO:teuthology.orchestra.run.smithi187.stdout: jq-1.6-7.el8.x86_64 2024-01-24T13:28:29.527 INFO:teuthology.orchestra.run.smithi187.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-01-24T13:28:29.527 INFO:teuthology.orchestra.run.smithi187.stdout: krb5-devel-1.18.2-26.el8_9.x86_64 2024-01-24T13:28:29.527 INFO:teuthology.orchestra.run.smithi187.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-01-24T13:28:29.527 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs-devel-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:29.527 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs2-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:29.527 INFO:teuthology.orchestra.run.smithi187.stdout: libcephsqlite-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:29.527 INFO:teuthology.orchestra.run.smithi187.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2024-01-24T13:28:29.527 INFO:teuthology.orchestra.run.smithi187.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-01-24T13:28:29.527 INFO:teuthology.orchestra.run.smithi187.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-24T13:28:29.527 INFO:teuthology.orchestra.run.smithi187.stdout: libpq-13.11-1.el8.x86_64 2024-01-24T13:28:29.527 INFO:teuthology.orchestra.run.smithi187.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-01-24T13:28:29.528 INFO:teuthology.orchestra.run.smithi187.stdout: librabbitmq-0.9.0-4.el8.x86_64 2024-01-24T13:28:29.528 INFO:teuthology.orchestra.run.smithi187.stdout: librados-devel-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:29.528 INFO:teuthology.orchestra.run.smithi187.stdout: libradosstriper1-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:29.528 INFO:teuthology.orchestra.run.smithi187.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-01-24T13:28:29.528 INFO:teuthology.orchestra.run.smithi187.stdout: librgw2-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:29.528 INFO:teuthology.orchestra.run.smithi187.stdout: libselinux-devel-2.9-5.el8.x86_64 2024-01-24T13:28:29.528 INFO:teuthology.orchestra.run.smithi187.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-01-24T13:28:29.528 INFO:teuthology.orchestra.run.smithi187.stdout: libtool-2.4.6-25.el8.x86_64 2024-01-24T13:28:29.528 INFO:teuthology.orchestra.run.smithi187.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-24T13:28:29.528 INFO:teuthology.orchestra.run.smithi187.stdout: libuuid-devel-2.32.1-43.el8.x86_64 2024-01-24T13:28:29.528 INFO:teuthology.orchestra.run.smithi187.stdout: libverto-devel-0.3.0-5.el8.x86_64 2024-01-24T13:28:29.528 INFO:teuthology.orchestra.run.smithi187.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-01-24T13:28:29.528 INFO:teuthology.orchestra.run.smithi187.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-24T13:28:29.528 INFO:teuthology.orchestra.run.smithi187.stdout: oniguruma-6.8.2-2.el8.x86_64 2024-01-24T13:28:29.529 INFO:teuthology.orchestra.run.smithi187.stdout: openblas-0.3.15-6.el8.x86_64 2024-01-24T13:28:29.529 INFO:teuthology.orchestra.run.smithi187.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-01-24T13:28:29.529 INFO:teuthology.orchestra.run.smithi187.stdout: openssl-devel-1:1.1.1k-12.el8_9.x86_64 2024-01-24T13:28:29.529 INFO:teuthology.orchestra.run.smithi187.stdout: pcre2-devel-10.32-2.el8.x86_64 2024-01-24T13:28:29.529 INFO:teuthology.orchestra.run.smithi187.stdout: pcre2-utf16-10.32-2.el8.x86_64 2024-01-24T13:28:29.529 INFO:teuthology.orchestra.run.smithi187.stdout: pcre2-utf32-10.32-2.el8.x86_64 2024-01-24T13:28:29.529 INFO:teuthology.orchestra.run.smithi187.stdout: postgresql-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-01-24T13:28:29.529 INFO:teuthology.orchestra.run.smithi187.stdout: postgresql-contrib-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-01-24T13:28:29.529 INFO:teuthology.orchestra.run.smithi187.stdout: postgresql-server-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-01-24T13:28:29.529 INFO:teuthology.orchestra.run.smithi187.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-01-24T13:28:29.529 INFO:teuthology.orchestra.run.smithi187.stdout: python3-babel-2.5.1-7.el8.noarch 2024-01-24T13:28:29.529 INFO:teuthology.orchestra.run.smithi187.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-24T13:28:29.529 INFO:teuthology.orchestra.run.smithi187.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-24T13:28:29.529 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-24T13:28:29.530 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-argparse-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:29.530 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-common-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:29.530 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cephfs-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:29.530 INFO:teuthology.orchestra.run.smithi187.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-24T13:28:29.530 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-24T13:28:29.530 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-24T13:28:29.530 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-24T13:28:29.530 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2024-01-24T13:28:29.530 INFO:teuthology.orchestra.run.smithi187.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-24T13:28:29.530 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-24T13:28:29.530 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-24T13:28:29.530 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-01-24T13:28:29.530 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-24T13:28:29.530 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-24T13:28:29.531 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-24T13:28:29.531 INFO:teuthology.orchestra.run.smithi187.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-24T13:28:29.531 INFO:teuthology.orchestra.run.smithi187.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-24T13:28:29.531 INFO:teuthology.orchestra.run.smithi187.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-24T13:28:29.531 INFO:teuthology.orchestra.run.smithi187.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-24T13:28:29.531 INFO:teuthology.orchestra.run.smithi187.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-24T13:28:29.531 INFO:teuthology.orchestra.run.smithi187.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-01-24T13:28:29.531 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-24T13:28:29.531 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-24T13:28:29.531 INFO:teuthology.orchestra.run.smithi187.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-24T13:28:29.531 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-24T13:28:29.531 INFO:teuthology.orchestra.run.smithi187.stdout: python3-portend-2.6-1.el8.noarch 2024-01-24T13:28:29.531 INFO:teuthology.orchestra.run.smithi187.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-24T13:28:29.532 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-24T13:28:29.532 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-24T13:28:29.532 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-24T13:28:29.532 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-24T13:28:29.532 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-24T13:28:29.532 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rados-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:29.532 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rbd-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:29.532 INFO:teuthology.orchestra.run.smithi187.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-24T13:28:29.532 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-24T13:28:29.532 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rgw-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:29.532 INFO:teuthology.orchestra.run.smithi187.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-24T13:28:29.532 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-24T13:28:29.532 INFO:teuthology.orchestra.run.smithi187.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-01-24T13:28:29.533 INFO:teuthology.orchestra.run.smithi187.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-24T13:28:29.533 INFO:teuthology.orchestra.run.smithi187.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-24T13:28:29.533 INFO:teuthology.orchestra.run.smithi187.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-24T13:28:29.533 INFO:teuthology.orchestra.run.smithi187.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-24T13:28:29.533 INFO:teuthology.orchestra.run.smithi187.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-24T13:28:29.533 INFO:teuthology.orchestra.run.smithi187.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-24T13:28:29.533 INFO:teuthology.orchestra.run.smithi187.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-24T13:28:29.533 INFO:teuthology.orchestra.run.smithi187.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-24T13:28:29.533 INFO:teuthology.orchestra.run.smithi187.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-24T13:28:29.533 INFO:teuthology.orchestra.run.smithi187.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-24T13:28:29.533 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-fuse-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:29.533 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:29.533 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-nbd-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:29.533 INFO:teuthology.orchestra.run.smithi187.stdout: socat-1.7.4.1-1.el8.x86_64 2024-01-24T13:28:29.533 INFO:teuthology.orchestra.run.smithi187.stdout: thrift-0.13.0-2.el8.x86_64 2024-01-24T13:28:29.534 INFO:teuthology.orchestra.run.smithi187.stdout: uuid-1.6.2-43.el8.x86_64 2024-01-24T13:28:29.534 INFO:teuthology.orchestra.run.smithi187.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-01-24T13:28:29.534 INFO:teuthology.orchestra.run.smithi187.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-01-24T13:28:29.534 INFO:teuthology.orchestra.run.smithi187.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2024-01-24T13:28:29.534 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-01-24T13:28:29.534 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-01-24T13:28:29.746 DEBUG:teuthology.parallel:result is None 2024-01-24T13:28:29.858 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : elfutils-libs-0.189-3.el8.x86_64 135/161 2024-01-24T13:28:30.275 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : iproute-6.2.0-5.el8_9.x86_64 136/161 2024-01-24T13:28:30.568 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : xfsdump-3.1.8-7.el8_9.x86_64 137/161 2024-01-24T13:28:30.747 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephfs-top-2:18.2.1-341.g4e3182a6.el8.noarch 138/161 2024-01-24T13:28:30.973 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-fuse-2:18.2.1-341.g4e3182a6.el8.x86_64 139/161 2024-01-24T13:28:31.268 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-nbd-2:18.2.1-341.g4e3182a6.el8.x86_64 140/161 2024-01-24T13:28:31.573 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : openssl-1:1.1.1k-12.el8_9.x86_64 141/161 2024-01-24T13:28:31.738 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : NetworkManager-initscripts-updown-1:1.40.16-13.e 142/161 2024-01-24T13:28:31.750 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-13.e 142/161 2024-01-24T13:28:31.972 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 143/161 2024-01-24T13:28:32.195 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flex-2.6.1-9.el8.x86_64 144/161 2024-01-24T13:28:32.283 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 144/161 2024-01-24T13:28:32.521 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : bison-3.0.4-10.el8.x86_64 145/161 2024-01-24T13:28:32.537 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 145/161 2024-01-24T13:28:32.734 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : elfutils-libs-0.186-1.el8.x86_64 146/161 2024-01-24T13:28:32.940 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 147/161 2024-01-24T13:28:33.117 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 148/161 2024-01-24T13:28:33.118 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 149/161 2024-01-24T13:28:33.466 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 149/161 2024-01-24T13:28:33.466 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 150/161 2024-01-24T13:28:33.708 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 150/161 2024-01-24T13:28:33.709 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 151/161 2024-01-24T13:28:33.978 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 151/161 2024-01-24T13:28:34.223 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 152/161 2024-01-24T13:28:34.394 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : util-linux-user-2.32.1-35.el8.x86_64 153/161 2024-01-24T13:28:34.631 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : util-linux-2.32.1-35.el8.x86_64 154/161 2024-01-24T13:28:34.631 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : libfdisk-2.32.1-35.el8.x86_64 155/161 2024-01-24T13:28:34.890 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libfdisk-2.32.1-35.el8.x86_64 155/161 2024-01-24T13:28:34.890 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : libmount-2.32.1-35.el8.x86_64 156/161 2024-01-24T13:28:35.132 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libmount-2.32.1-35.el8.x86_64 156/161 2024-01-24T13:28:35.132 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : libblkid-2.32.1-35.el8.x86_64 157/161 2024-01-24T13:28:35.357 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libblkid-2.32.1-35.el8.x86_64 157/161 2024-01-24T13:28:35.357 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : libuuid-2.32.1-35.el8.x86_64 158/161 2024-01-24T13:28:35.589 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libuuid-2.32.1-35.el8.x86_64 158/161 2024-01-24T13:28:35.590 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : libsmartcols-2.32.1-35.el8.x86_64 159/161 2024-01-24T13:28:35.831 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libsmartcols-2.32.1-35.el8.x86_64 159/161 2024-01-24T13:28:36.007 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : elfutils-libelf-0.186-1.el8.x86_64 160/161 2024-01-24T13:28:36.007 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 161/161 2024-01-24T13:28:37.309 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 161/161 2024-01-24T13:28:37.309 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-2:18.2.1-341.g4e3182a6.el8.x86_64 1/161 2024-01-24T13:28:37.309 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-base-2:18.2.1-341.g4e3182a6.el8.x86_64 2/161 2024-01-24T13:28:37.309 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-common-2:18.2.1-341.g4e3182a6.el8.x86_64 3/161 2024-01-24T13:28:37.309 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-fuse-2:18.2.1-341.g4e3182a6.el8.x86_64 4/161 2024-01-24T13:28:37.309 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-immutable-object-cache-2:18.2.1-341.g4e3182 5/161 2024-01-24T13:28:37.309 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mds-2:18.2.1-341.g4e3182a6.el8.x86_64 6/161 2024-01-24T13:28:37.309 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-2:18.2.1-341.g4e3182a6.el8.x86_64 7/161 2024-01-24T13:28:37.309 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mon-2:18.2.1-341.g4e3182a6.el8.x86_64 8/161 2024-01-24T13:28:37.309 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-osd-2:18.2.1-341.g4e3182a6.el8.x86_64 9/161 2024-01-24T13:28:37.310 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-radosgw-2:18.2.1-341.g4e3182a6.el8.x86_64 10/161 2024-01-24T13:28:37.310 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-selinux-2:18.2.1-341.g4e3182a6.el8.x86_64 11/161 2024-01-24T13:28:37.310 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-test-2:18.2.1-341.g4e3182a6.el8.x86_64 12/161 2024-01-24T13:28:37.310 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephfs-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 13/161 2024-01-24T13:28:37.310 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs-devel-2:18.2.1-341.g4e3182a6.el8.x86_6 14/161 2024-01-24T13:28:37.310 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs2-2:18.2.1-341.g4e3182a6.el8.x86_64 15/161 2024-01-24T13:28:37.310 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephsqlite-2:18.2.1-341.g4e3182a6.el8.x86_64 16/161 2024-01-24T13:28:37.310 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados-devel-2:18.2.1-341.g4e3182a6.el8.x86_64 17/161 2024-01-24T13:28:37.310 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libradosstriper1-2:18.2.1-341.g4e3182a6.el8.x86_ 18/161 2024-01-24T13:28:37.310 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librgw2-2:18.2.1-341.g4e3182a6.el8.x86_64 19/161 2024-01-24T13:28:37.310 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-argparse-2:18.2.1-341.g4e3182a6.el8 20/161 2024-01-24T13:28:37.310 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-common-2:18.2.1-341.g4e3182a6.el8.x 21/161 2024-01-24T13:28:37.311 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cephfs-2:18.2.1-341.g4e3182a6.el8.x86_64 22/161 2024-01-24T13:28:37.311 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rados-2:18.2.1-341.g4e3182a6.el8.x86_64 23/161 2024-01-24T13:28:37.311 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rbd-2:18.2.1-341.g4e3182a6.el8.x86_64 24/161 2024-01-24T13:28:37.311 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rgw-2:18.2.1-341.g4e3182a6.el8.x86_64 25/161 2024-01-24T13:28:37.311 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-fuse-2:18.2.1-341.g4e3182a6.el8.x86_64 26/161 2024-01-24T13:28:37.311 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 27/161 2024-01-24T13:28:37.311 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-nbd-2:18.2.1-341.g4e3182a6.el8.x86_64 28/161 2024-01-24T13:28:37.311 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-grafana-dashboards-2:18.2.1-341.g4e3182a6.e 29/161 2024-01-24T13:28:37.311 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-cephadm-2:18.2.1-341.g4e3182a6.el8.noar 30/161 2024-01-24T13:28:37.311 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-dashboard-2:18.2.1-341.g4e3182a6.el8.no 31/161 2024-01-24T13:28:37.311 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.1-341.g4e31 32/161 2024-01-24T13:28:37.311 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-modules-core-2:18.2.1-341.g4e3182a6.el8 33/161 2024-01-24T13:28:37.312 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-rook-2:18.2.1-341.g4e3182a6.el8.noarch 34/161 2024-01-24T13:28:37.312 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-prometheus-alerts-2:18.2.1-341.g4e3182a6.el 35/161 2024-01-24T13:28:37.312 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-volume-2:18.2.1-341.g4e3182a6.el8.noarch 36/161 2024-01-24T13:28:37.312 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephadm-2:18.2.1-341.g4e3182a6.el8.noarch 37/161 2024-01-24T13:28:37.312 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephfs-top-2:18.2.1-341.g4e3182a6.el8.noarch 38/161 2024-01-24T13:28:37.312 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/161 2024-01-24T13:28:37.312 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/161 2024-01-24T13:28:37.312 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/161 2024-01-24T13:28:37.312 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/161 2024-01-24T13:28:37.312 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/161 2024-01-24T13:28:37.312 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/161 2024-01-24T13:28:37.313 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/161 2024-01-24T13:28:37.313 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/161 2024-01-24T13:28:37.313 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/161 2024-01-24T13:28:37.313 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/161 2024-01-24T13:28:37.313 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/161 2024-01-24T13:28:37.313 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/161 2024-01-24T13:28:37.313 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/161 2024-01-24T13:28:37.313 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/161 2024-01-24T13:28:37.313 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/161 2024-01-24T13:28:37.313 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/161 2024-01-24T13:28:37.313 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/161 2024-01-24T13:28:37.313 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/161 2024-01-24T13:28:37.314 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/161 2024-01-24T13:28:37.314 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/161 2024-01-24T13:28:37.314 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/161 2024-01-24T13:28:37.314 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/161 2024-01-24T13:28:37.314 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/161 2024-01-24T13:28:37.314 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/161 2024-01-24T13:28:37.314 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/161 2024-01-24T13:28:37.314 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/161 2024-01-24T13:28:37.314 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/161 2024-01-24T13:28:37.314 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/161 2024-01-24T13:28:37.314 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/161 2024-01-24T13:28:37.314 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/161 2024-01-24T13:28:37.314 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/161 2024-01-24T13:28:37.315 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/161 2024-01-24T13:28:37.315 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/161 2024-01-24T13:28:37.315 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 72/161 2024-01-24T13:28:37.315 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 73/161 2024-01-24T13:28:37.315 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 74/161 2024-01-24T13:28:37.315 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 75/161 2024-01-24T13:28:37.315 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 76/161 2024-01-24T13:28:37.315 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 77/161 2024-01-24T13:28:37.315 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/161 2024-01-24T13:28:37.315 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flex-2.6.1-9.el8.x86_64 79/161 2024-01-24T13:28:37.315 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 80/161 2024-01-24T13:28:37.316 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 81/161 2024-01-24T13:28:37.316 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 82/161 2024-01-24T13:28:37.316 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 83/161 2024-01-24T13:28:37.316 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 84/161 2024-01-24T13:28:37.316 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 85/161 2024-01-24T13:28:37.316 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 86/161 2024-01-24T13:28:37.316 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 87/161 2024-01-24T13:28:37.316 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 88/161 2024-01-24T13:28:37.316 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 89/161 2024-01-24T13:28:37.316 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 90/161 2024-01-24T13:28:37.316 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 91/161 2024-01-24T13:28:37.317 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 92/161 2024-01-24T13:28:37.317 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 93/161 2024-01-24T13:28:37.317 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 94/161 2024-01-24T13:28:37.317 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : automake-1.16.1-8.el8.noarch 95/161 2024-01-24T13:28:37.317 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libpq-13.11-1.el8.x86_64 96/161 2024-01-24T13:28:37.317 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : jq-1.6-7.el8.x86_64 97/161 2024-01-24T13:28:37.317 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 98/161 2024-01-24T13:28:37.317 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : postgresql-10.23-3.module+el8.9.0+20831+78489779 99/161 2024-01-24T13:28:37.317 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : postgresql-contrib-10.23-3.module+el8.9.0+20831+ 100/161 2024-01-24T13:28:37.317 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : postgresql-server-10.23-3.module+el8.9.0+20831+7 101/161 2024-01-24T13:28:37.317 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 102/161 2024-01-24T13:28:37.318 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 103/161 2024-01-24T13:28:37.318 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 104/161 2024-01-24T13:28:37.318 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 105/161 2024-01-24T13:28:37.318 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 106/161 2024-01-24T13:28:37.318 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 107/161 2024-01-24T13:28:37.318 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 108/161 2024-01-24T13:28:37.318 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 109/161 2024-01-24T13:28:37.318 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 110/161 2024-01-24T13:28:37.318 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/161 2024-01-24T13:28:37.318 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 112/161 2024-01-24T13:28:37.318 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 113/161 2024-01-24T13:28:37.318 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 114/161 2024-01-24T13:28:37.319 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 115/161 2024-01-24T13:28:37.319 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 116/161 2024-01-24T13:28:37.319 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 117/161 2024-01-24T13:28:37.319 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 118/161 2024-01-24T13:28:37.319 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 119/161 2024-01-24T13:28:37.319 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 120/161 2024-01-24T13:28:37.319 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libuuid-devel-2.32.1-43.el8.x86_64 121/161 2024-01-24T13:28:37.319 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 122/161 2024-01-24T13:28:37.319 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 123/161 2024-01-24T13:28:37.319 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 124/161 2024-01-24T13:28:37.319 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : elfutils-libelf-devel-0.189-3.el8.x86_64 125/161 2024-01-24T13:28:37.319 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 126/161 2024-01-24T13:28:37.320 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : krb5-devel-1.18.2-26.el8_9.x86_64 127/161 2024-01-24T13:28:37.320 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8_9.x86_64 128/161 2024-01-24T13:28:37.320 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-13.e 129/161 2024-01-24T13:28:37.320 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-2:18.2.1-341.g4e3182a6.el8.x86_64 130/161 2024-01-24T13:28:37.320 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 131/161 2024-01-24T13:28:37.320 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-2:18.2.1-341.g4e3182a6.el8.x86_64 132/161 2024-01-24T13:28:37.320 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 133/161 2024-01-24T13:28:37.320 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : elfutils-libelf-0.189-3.el8.x86_64 134/161 2024-01-24T13:28:37.320 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : elfutils-libelf-0.186-1.el8.x86_64 135/161 2024-01-24T13:28:37.320 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : elfutils-libs-0.189-3.el8.x86_64 136/161 2024-01-24T13:28:37.320 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : elfutils-libs-0.186-1.el8.x86_64 137/161 2024-01-24T13:28:37.321 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libmount-2.32.1-43.el8.x86_64 138/161 2024-01-24T13:28:37.321 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libmount-2.32.1-35.el8.x86_64 139/161 2024-01-24T13:28:37.321 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libsmartcols-2.32.1-43.el8.x86_64 140/161 2024-01-24T13:28:37.321 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libsmartcols-2.32.1-35.el8.x86_64 141/161 2024-01-24T13:28:37.321 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libuuid-2.32.1-43.el8.x86_64 142/161 2024-01-24T13:28:37.321 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libuuid-2.32.1-35.el8.x86_64 143/161 2024-01-24T13:28:37.321 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : util-linux-2.32.1-43.el8.x86_64 144/161 2024-01-24T13:28:37.321 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : util-linux-2.32.1-35.el8.x86_64 145/161 2024-01-24T13:28:37.321 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : util-linux-user-2.32.1-43.el8.x86_64 146/161 2024-01-24T13:28:37.321 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : util-linux-user-2.32.1-35.el8.x86_64 147/161 2024-01-24T13:28:37.321 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : iproute-6.2.0-5.el8_9.x86_64 148/161 2024-01-24T13:28:37.322 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 149/161 2024-01-24T13:28:37.322 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libblkid-2.32.1-43.el8.x86_64 150/161 2024-01-24T13:28:37.322 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libblkid-2.32.1-35.el8.x86_64 151/161 2024-01-24T13:28:37.322 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libfdisk-2.32.1-43.el8.x86_64 152/161 2024-01-24T13:28:37.322 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libfdisk-2.32.1-35.el8.x86_64 153/161 2024-01-24T13:28:37.322 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 154/161 2024-01-24T13:28:37.322 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 155/161 2024-01-24T13:28:37.322 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openssl-1:1.1.1k-12.el8_9.x86_64 156/161 2024-01-24T13:28:37.322 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 157/161 2024-01-24T13:28:37.322 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8_9.x86_64 158/161 2024-01-24T13:28:37.322 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 159/161 2024-01-24T13:28:37.322 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xfsdump-3.1.8-7.el8_9.x86_64 160/161 2024-01-24T13:28:38.179 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 161/161 2024-01-24T13:28:38.179 INFO:teuthology.orchestra.run.smithi163.stdout:Installed products updated. 2024-01-24T13:28:38.772 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-01-24T13:28:38.772 INFO:teuthology.orchestra.run.smithi163.stdout:Upgraded: 2024-01-24T13:28:38.772 INFO:teuthology.orchestra.run.smithi163.stdout: elfutils-libelf-0.189-3.el8.x86_64 2024-01-24T13:28:38.773 INFO:teuthology.orchestra.run.smithi163.stdout: elfutils-libs-0.189-3.el8.x86_64 2024-01-24T13:28:38.773 INFO:teuthology.orchestra.run.smithi163.stdout: iproute-6.2.0-5.el8_9.x86_64 2024-01-24T13:28:38.773 INFO:teuthology.orchestra.run.smithi163.stdout: libblkid-2.32.1-43.el8.x86_64 2024-01-24T13:28:38.773 INFO:teuthology.orchestra.run.smithi163.stdout: libfdisk-2.32.1-43.el8.x86_64 2024-01-24T13:28:38.773 INFO:teuthology.orchestra.run.smithi163.stdout: libmount-2.32.1-43.el8.x86_64 2024-01-24T13:28:38.773 INFO:teuthology.orchestra.run.smithi163.stdout: librados2-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:38.773 INFO:teuthology.orchestra.run.smithi163.stdout: librbd1-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:38.773 INFO:teuthology.orchestra.run.smithi163.stdout: libsmartcols-2.32.1-43.el8.x86_64 2024-01-24T13:28:38.773 INFO:teuthology.orchestra.run.smithi163.stdout: libuuid-2.32.1-43.el8.x86_64 2024-01-24T13:28:38.773 INFO:teuthology.orchestra.run.smithi163.stdout: openssl-1:1.1.1k-12.el8_9.x86_64 2024-01-24T13:28:38.773 INFO:teuthology.orchestra.run.smithi163.stdout: openssl-libs-1:1.1.1k-12.el8_9.x86_64 2024-01-24T13:28:38.773 INFO:teuthology.orchestra.run.smithi163.stdout: util-linux-2.32.1-43.el8.x86_64 2024-01-24T13:28:38.773 INFO:teuthology.orchestra.run.smithi163.stdout: util-linux-user-2.32.1-43.el8.x86_64 2024-01-24T13:28:38.774 INFO:teuthology.orchestra.run.smithi163.stdout: xfsdump-3.1.8-7.el8_9.x86_64 2024-01-24T13:28:38.774 INFO:teuthology.orchestra.run.smithi163.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-01-24T13:28:38.774 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-01-24T13:28:38.774 INFO:teuthology.orchestra.run.smithi163.stdout: NetworkManager-initscripts-updown-1:1.40.16-13.el8_9.noarch 2024-01-24T13:28:38.774 INFO:teuthology.orchestra.run.smithi163.stdout: automake-1.16.1-8.el8.noarch 2024-01-24T13:28:38.774 INFO:teuthology.orchestra.run.smithi163.stdout: bison-3.0.4-10.el8.x86_64 2024-01-24T13:28:38.774 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:38.774 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-base-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:38.774 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-common-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:38.774 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-fuse-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:38.774 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-grafana-dashboards-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T13:28:38.774 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-immutable-object-cache-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:38.774 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mds-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:38.775 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:38.775 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-cephadm-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T13:28:38.775 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-dashboard-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T13:28:38.775 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-diskprediction-local-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T13:28:38.775 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-modules-core-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T13:28:38.775 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-rook-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T13:28:38.775 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mon-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:38.775 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-osd-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:38.775 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-prometheus-alerts-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T13:28:38.775 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-radosgw-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:38.775 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-selinux-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:38.775 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-test-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:38.775 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-volume-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T13:28:38.776 INFO:teuthology.orchestra.run.smithi163.stdout: cephadm-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T13:28:38.776 INFO:teuthology.orchestra.run.smithi163.stdout: cephfs-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:38.776 INFO:teuthology.orchestra.run.smithi163.stdout: cephfs-top-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T13:28:38.776 INFO:teuthology.orchestra.run.smithi163.stdout: elfutils-libelf-devel-0.189-3.el8.x86_64 2024-01-24T13:28:38.776 INFO:teuthology.orchestra.run.smithi163.stdout: flex-2.6.1-9.el8.x86_64 2024-01-24T13:28:38.776 INFO:teuthology.orchestra.run.smithi163.stdout: fuse-2.9.7-17.el8.x86_64 2024-01-24T13:28:38.776 INFO:teuthology.orchestra.run.smithi163.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-24T13:28:38.776 INFO:teuthology.orchestra.run.smithi163.stdout: jq-1.6-7.el8.x86_64 2024-01-24T13:28:38.776 INFO:teuthology.orchestra.run.smithi163.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-01-24T13:28:38.776 INFO:teuthology.orchestra.run.smithi163.stdout: krb5-devel-1.18.2-26.el8_9.x86_64 2024-01-24T13:28:38.776 INFO:teuthology.orchestra.run.smithi163.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-01-24T13:28:38.776 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs-devel-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:38.776 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs2-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:38.777 INFO:teuthology.orchestra.run.smithi163.stdout: libcephsqlite-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:38.777 INFO:teuthology.orchestra.run.smithi163.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2024-01-24T13:28:38.777 INFO:teuthology.orchestra.run.smithi163.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-01-24T13:28:38.777 INFO:teuthology.orchestra.run.smithi163.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-24T13:28:38.777 INFO:teuthology.orchestra.run.smithi163.stdout: libpq-13.11-1.el8.x86_64 2024-01-24T13:28:38.777 INFO:teuthology.orchestra.run.smithi163.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-01-24T13:28:38.777 INFO:teuthology.orchestra.run.smithi163.stdout: librabbitmq-0.9.0-4.el8.x86_64 2024-01-24T13:28:38.777 INFO:teuthology.orchestra.run.smithi163.stdout: librados-devel-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:38.777 INFO:teuthology.orchestra.run.smithi163.stdout: libradosstriper1-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:38.777 INFO:teuthology.orchestra.run.smithi163.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-01-24T13:28:38.777 INFO:teuthology.orchestra.run.smithi163.stdout: librgw2-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:38.777 INFO:teuthology.orchestra.run.smithi163.stdout: libselinux-devel-2.9-5.el8.x86_64 2024-01-24T13:28:38.777 INFO:teuthology.orchestra.run.smithi163.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-01-24T13:28:38.778 INFO:teuthology.orchestra.run.smithi163.stdout: libtool-2.4.6-25.el8.x86_64 2024-01-24T13:28:38.778 INFO:teuthology.orchestra.run.smithi163.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-24T13:28:38.778 INFO:teuthology.orchestra.run.smithi163.stdout: libuuid-devel-2.32.1-43.el8.x86_64 2024-01-24T13:28:38.778 INFO:teuthology.orchestra.run.smithi163.stdout: libverto-devel-0.3.0-5.el8.x86_64 2024-01-24T13:28:38.778 INFO:teuthology.orchestra.run.smithi163.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-01-24T13:28:38.778 INFO:teuthology.orchestra.run.smithi163.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-24T13:28:38.778 INFO:teuthology.orchestra.run.smithi163.stdout: oniguruma-6.8.2-2.el8.x86_64 2024-01-24T13:28:38.778 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-0.3.15-6.el8.x86_64 2024-01-24T13:28:38.778 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-01-24T13:28:38.778 INFO:teuthology.orchestra.run.smithi163.stdout: openssl-devel-1:1.1.1k-12.el8_9.x86_64 2024-01-24T13:28:38.778 INFO:teuthology.orchestra.run.smithi163.stdout: pcre2-devel-10.32-2.el8.x86_64 2024-01-24T13:28:38.778 INFO:teuthology.orchestra.run.smithi163.stdout: pcre2-utf16-10.32-2.el8.x86_64 2024-01-24T13:28:38.778 INFO:teuthology.orchestra.run.smithi163.stdout: pcre2-utf32-10.32-2.el8.x86_64 2024-01-24T13:28:38.779 INFO:teuthology.orchestra.run.smithi163.stdout: postgresql-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-01-24T13:28:38.779 INFO:teuthology.orchestra.run.smithi163.stdout: postgresql-contrib-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-01-24T13:28:38.779 INFO:teuthology.orchestra.run.smithi163.stdout: postgresql-server-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-01-24T13:28:38.779 INFO:teuthology.orchestra.run.smithi163.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-01-24T13:28:38.779 INFO:teuthology.orchestra.run.smithi163.stdout: python3-babel-2.5.1-7.el8.noarch 2024-01-24T13:28:38.779 INFO:teuthology.orchestra.run.smithi163.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-24T13:28:38.779 INFO:teuthology.orchestra.run.smithi163.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-24T13:28:38.779 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-24T13:28:38.779 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-argparse-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:38.780 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-common-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:38.780 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cephfs-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:38.780 INFO:teuthology.orchestra.run.smithi163.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-24T13:28:38.780 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-24T13:28:38.780 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-24T13:28:38.780 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-24T13:28:38.780 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2024-01-24T13:28:38.780 INFO:teuthology.orchestra.run.smithi163.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-24T13:28:38.780 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-24T13:28:38.780 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-24T13:28:38.780 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-01-24T13:28:38.781 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-24T13:28:38.781 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-24T13:28:38.781 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-24T13:28:38.781 INFO:teuthology.orchestra.run.smithi163.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-24T13:28:38.781 INFO:teuthology.orchestra.run.smithi163.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-24T13:28:38.781 INFO:teuthology.orchestra.run.smithi163.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-24T13:28:38.781 INFO:teuthology.orchestra.run.smithi163.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-24T13:28:38.781 INFO:teuthology.orchestra.run.smithi163.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-24T13:28:38.781 INFO:teuthology.orchestra.run.smithi163.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-01-24T13:28:38.781 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-24T13:28:38.781 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-24T13:28:38.781 INFO:teuthology.orchestra.run.smithi163.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-24T13:28:38.781 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-24T13:28:38.782 INFO:teuthology.orchestra.run.smithi163.stdout: python3-portend-2.6-1.el8.noarch 2024-01-24T13:28:38.782 INFO:teuthology.orchestra.run.smithi163.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-24T13:28:38.782 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-24T13:28:38.782 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-24T13:28:38.782 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-24T13:28:38.782 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-24T13:28:38.782 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-24T13:28:38.782 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rados-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:38.782 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rbd-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:38.782 INFO:teuthology.orchestra.run.smithi163.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-24T13:28:38.782 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-24T13:28:38.782 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rgw-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:38.782 INFO:teuthology.orchestra.run.smithi163.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-24T13:28:38.782 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-24T13:28:38.783 INFO:teuthology.orchestra.run.smithi163.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-01-24T13:28:38.783 INFO:teuthology.orchestra.run.smithi163.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-24T13:28:38.783 INFO:teuthology.orchestra.run.smithi163.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-24T13:28:38.783 INFO:teuthology.orchestra.run.smithi163.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-24T13:28:38.783 INFO:teuthology.orchestra.run.smithi163.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-24T13:28:38.783 INFO:teuthology.orchestra.run.smithi163.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-24T13:28:38.783 INFO:teuthology.orchestra.run.smithi163.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-24T13:28:38.783 INFO:teuthology.orchestra.run.smithi163.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-24T13:28:38.783 INFO:teuthology.orchestra.run.smithi163.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-24T13:28:38.783 INFO:teuthology.orchestra.run.smithi163.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-24T13:28:38.783 INFO:teuthology.orchestra.run.smithi163.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-24T13:28:38.783 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-fuse-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:38.783 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:38.783 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-nbd-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:38.784 INFO:teuthology.orchestra.run.smithi163.stdout: socat-1.7.4.1-1.el8.x86_64 2024-01-24T13:28:38.784 INFO:teuthology.orchestra.run.smithi163.stdout: thrift-0.13.0-2.el8.x86_64 2024-01-24T13:28:38.784 INFO:teuthology.orchestra.run.smithi163.stdout: uuid-1.6.2-43.el8.x86_64 2024-01-24T13:28:38.784 INFO:teuthology.orchestra.run.smithi163.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-01-24T13:28:38.784 INFO:teuthology.orchestra.run.smithi163.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-01-24T13:28:38.787 INFO:teuthology.orchestra.run.smithi163.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2024-01-24T13:28:38.787 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-01-24T13:28:38.787 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-01-24T13:28:39.030 DEBUG:teuthology.parallel:result is None 2024-01-24T13:28:49.313 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 161/161 2024-01-24T13:28:49.313 INFO:teuthology.orchestra.run.smithi029.stdout:Installed products updated. 2024-01-24T13:28:49.981 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:28:49.981 INFO:teuthology.orchestra.run.smithi029.stdout:Upgraded: 2024-01-24T13:28:49.982 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-libelf-0.189-3.el8.x86_64 2024-01-24T13:28:49.982 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-libs-0.189-3.el8.x86_64 2024-01-24T13:28:49.982 INFO:teuthology.orchestra.run.smithi029.stdout: iproute-6.2.0-5.el8_9.x86_64 2024-01-24T13:28:49.982 INFO:teuthology.orchestra.run.smithi029.stdout: libblkid-2.32.1-43.el8.x86_64 2024-01-24T13:28:49.982 INFO:teuthology.orchestra.run.smithi029.stdout: libfdisk-2.32.1-43.el8.x86_64 2024-01-24T13:28:49.982 INFO:teuthology.orchestra.run.smithi029.stdout: libmount-2.32.1-43.el8.x86_64 2024-01-24T13:28:49.982 INFO:teuthology.orchestra.run.smithi029.stdout: librados2-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:49.982 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:49.982 INFO:teuthology.orchestra.run.smithi029.stdout: libsmartcols-2.32.1-43.el8.x86_64 2024-01-24T13:28:49.982 INFO:teuthology.orchestra.run.smithi029.stdout: libuuid-2.32.1-43.el8.x86_64 2024-01-24T13:28:49.982 INFO:teuthology.orchestra.run.smithi029.stdout: openssl-1:1.1.1k-12.el8_9.x86_64 2024-01-24T13:28:49.982 INFO:teuthology.orchestra.run.smithi029.stdout: openssl-libs-1:1.1.1k-12.el8_9.x86_64 2024-01-24T13:28:49.983 INFO:teuthology.orchestra.run.smithi029.stdout: util-linux-2.32.1-43.el8.x86_64 2024-01-24T13:28:49.983 INFO:teuthology.orchestra.run.smithi029.stdout: util-linux-user-2.32.1-43.el8.x86_64 2024-01-24T13:28:49.983 INFO:teuthology.orchestra.run.smithi029.stdout: xfsdump-3.1.8-7.el8_9.x86_64 2024-01-24T13:28:49.983 INFO:teuthology.orchestra.run.smithi029.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-01-24T13:28:49.983 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-01-24T13:28:49.983 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-initscripts-updown-1:1.40.16-13.el8_9.noarch 2024-01-24T13:28:49.983 INFO:teuthology.orchestra.run.smithi029.stdout: automake-1.16.1-8.el8.noarch 2024-01-24T13:28:49.983 INFO:teuthology.orchestra.run.smithi029.stdout: bison-3.0.4-10.el8.x86_64 2024-01-24T13:28:49.983 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:49.983 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:49.983 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:49.983 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:49.984 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T13:28:49.984 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:49.984 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:49.984 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:49.984 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T13:28:49.984 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T13:28:49.984 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T13:28:49.984 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T13:28:49.984 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T13:28:49.984 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:49.984 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:49.984 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T13:28:49.984 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:49.985 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:49.985 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:49.985 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-volume-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T13:28:49.985 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T13:28:49.985 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:49.985 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-top-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T13:28:49.985 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-libelf-devel-0.189-3.el8.x86_64 2024-01-24T13:28:49.985 INFO:teuthology.orchestra.run.smithi029.stdout: flex-2.6.1-9.el8.x86_64 2024-01-24T13:28:49.985 INFO:teuthology.orchestra.run.smithi029.stdout: fuse-2.9.7-17.el8.x86_64 2024-01-24T13:28:49.985 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-24T13:28:49.985 INFO:teuthology.orchestra.run.smithi029.stdout: jq-1.6-7.el8.x86_64 2024-01-24T13:28:49.985 INFO:teuthology.orchestra.run.smithi029.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-01-24T13:28:49.985 INFO:teuthology.orchestra.run.smithi029.stdout: krb5-devel-1.18.2-26.el8_9.x86_64 2024-01-24T13:28:49.985 INFO:teuthology.orchestra.run.smithi029.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-01-24T13:28:49.986 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:49.986 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:49.986 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:49.986 INFO:teuthology.orchestra.run.smithi029.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2024-01-24T13:28:49.986 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-01-24T13:28:49.986 INFO:teuthology.orchestra.run.smithi029.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-24T13:28:49.986 INFO:teuthology.orchestra.run.smithi029.stdout: libpq-13.11-1.el8.x86_64 2024-01-24T13:28:49.986 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-01-24T13:28:49.986 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq-0.9.0-4.el8.x86_64 2024-01-24T13:28:49.986 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:49.986 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:49.986 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-01-24T13:28:49.986 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:49.987 INFO:teuthology.orchestra.run.smithi029.stdout: libselinux-devel-2.9-5.el8.x86_64 2024-01-24T13:28:49.987 INFO:teuthology.orchestra.run.smithi029.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-01-24T13:28:49.987 INFO:teuthology.orchestra.run.smithi029.stdout: libtool-2.4.6-25.el8.x86_64 2024-01-24T13:28:49.987 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-24T13:28:49.987 INFO:teuthology.orchestra.run.smithi029.stdout: libuuid-devel-2.32.1-43.el8.x86_64 2024-01-24T13:28:49.987 INFO:teuthology.orchestra.run.smithi029.stdout: libverto-devel-0.3.0-5.el8.x86_64 2024-01-24T13:28:49.987 INFO:teuthology.orchestra.run.smithi029.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-01-24T13:28:49.987 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-24T13:28:49.987 INFO:teuthology.orchestra.run.smithi029.stdout: oniguruma-6.8.2-2.el8.x86_64 2024-01-24T13:28:49.987 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-0.3.15-6.el8.x86_64 2024-01-24T13:28:49.987 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-01-24T13:28:49.987 INFO:teuthology.orchestra.run.smithi029.stdout: openssl-devel-1:1.1.1k-12.el8_9.x86_64 2024-01-24T13:28:49.987 INFO:teuthology.orchestra.run.smithi029.stdout: pcre2-devel-10.32-2.el8.x86_64 2024-01-24T13:28:49.987 INFO:teuthology.orchestra.run.smithi029.stdout: pcre2-utf16-10.32-2.el8.x86_64 2024-01-24T13:28:49.988 INFO:teuthology.orchestra.run.smithi029.stdout: pcre2-utf32-10.32-2.el8.x86_64 2024-01-24T13:28:49.988 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-01-24T13:28:49.988 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-contrib-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-01-24T13:28:49.988 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-server-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-01-24T13:28:49.988 INFO:teuthology.orchestra.run.smithi029.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-01-24T13:28:49.988 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel-2.5.1-7.el8.noarch 2024-01-24T13:28:49.988 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-24T13:28:49.988 INFO:teuthology.orchestra.run.smithi029.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-24T13:28:49.988 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-24T13:28:49.988 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:49.988 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:49.988 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:49.988 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-24T13:28:49.988 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-24T13:28:49.989 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-24T13:28:49.989 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-24T13:28:49.989 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2024-01-24T13:28:49.989 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-24T13:28:49.989 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-24T13:28:49.989 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-24T13:28:49.989 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-01-24T13:28:49.989 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-24T13:28:49.989 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-24T13:28:49.989 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-24T13:28:49.989 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-24T13:28:49.989 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-24T13:28:49.989 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-24T13:28:49.990 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-24T13:28:49.990 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-24T13:28:49.990 INFO:teuthology.orchestra.run.smithi029.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-01-24T13:28:49.990 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-24T13:28:49.990 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-24T13:28:49.990 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-24T13:28:49.990 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-24T13:28:49.990 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend-2.6-1.el8.noarch 2024-01-24T13:28:49.990 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-24T13:28:49.990 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-24T13:28:49.990 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-24T13:28:49.990 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-24T13:28:49.990 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-24T13:28:49.990 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-24T13:28:49.991 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:49.991 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:49.991 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-24T13:28:49.991 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-24T13:28:49.991 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:49.991 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-24T13:28:49.991 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-24T13:28:49.991 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-01-24T13:28:49.991 INFO:teuthology.orchestra.run.smithi029.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-24T13:28:49.991 INFO:teuthology.orchestra.run.smithi029.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-24T13:28:49.991 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-24T13:28:49.991 INFO:teuthology.orchestra.run.smithi029.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-24T13:28:49.991 INFO:teuthology.orchestra.run.smithi029.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-24T13:28:49.991 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-24T13:28:49.991 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-24T13:28:49.992 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-24T13:28:49.992 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-24T13:28:49.992 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-24T13:28:49.992 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:49.992 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:49.992 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T13:28:49.992 INFO:teuthology.orchestra.run.smithi029.stdout: socat-1.7.4.1-1.el8.x86_64 2024-01-24T13:28:49.992 INFO:teuthology.orchestra.run.smithi029.stdout: thrift-0.13.0-2.el8.x86_64 2024-01-24T13:28:49.992 INFO:teuthology.orchestra.run.smithi029.stdout: uuid-1.6.2-43.el8.x86_64 2024-01-24T13:28:49.992 INFO:teuthology.orchestra.run.smithi029.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-01-24T13:28:49.992 INFO:teuthology.orchestra.run.smithi029.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-01-24T13:28:49.993 INFO:teuthology.orchestra.run.smithi029.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2024-01-24T13:28:49.993 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:28:49.993 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-24T13:28:50.256 DEBUG:teuthology.parallel:result is None 2024-01-24T13:28:50.257 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4e3182a65800aed6d4aa2f6b86da3cfd72188134 2024-01-24T13:28:50.477 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-24T13:28:50.507 INFO:teuthology.orchestra.run.smithi029.stdout:18.2.1-341.g4e3182a6.el8 2024-01-24T13:28:50.508 INFO:teuthology.packaging:The installed version of ceph is 18.2.1-341.g4e3182a6.el8 2024-01-24T13:28:50.508 INFO:teuthology.task.install:The correct ceph version 18.2.1-341.g4e3182a6 is installed. 2024-01-24T13:28:50.510 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4e3182a65800aed6d4aa2f6b86da3cfd72188134 2024-01-24T13:28:50.645 DEBUG:teuthology.orchestra.run.smithi163:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-24T13:28:50.675 INFO:teuthology.orchestra.run.smithi163.stdout:18.2.1-341.g4e3182a6.el8 2024-01-24T13:28:50.675 INFO:teuthology.packaging:The installed version of ceph is 18.2.1-341.g4e3182a6.el8 2024-01-24T13:28:50.675 INFO:teuthology.task.install:The correct ceph version 18.2.1-341.g4e3182a6 is installed. 2024-01-24T13:28:50.677 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4e3182a65800aed6d4aa2f6b86da3cfd72188134 2024-01-24T13:28:50.898 DEBUG:teuthology.orchestra.run.smithi187:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-24T13:28:50.927 INFO:teuthology.orchestra.run.smithi187.stdout:18.2.1-341.g4e3182a6.el8 2024-01-24T13:28:50.927 INFO:teuthology.packaging:The installed version of ceph is 18.2.1-341.g4e3182a6.el8 2024-01-24T13:28:50.927 INFO:teuthology.task.install:The correct ceph version 18.2.1-341.g4e3182a6 is installed. 2024-01-24T13:28:50.929 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-24T13:28:50.929 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-24T13:28:50.929 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-24T13:28:50.981 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-01-24T13:28:50.982 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-24T13:28:51.030 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-01-24T13:28:51.030 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-24T13:28:51.078 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-24T13:28:51.079 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-24T13:28:51.079 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/daemon-helper 2024-01-24T13:28:51.114 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-24T13:28:51.188 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-01-24T13:28:51.188 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/daemon-helper 2024-01-24T13:28:51.220 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-24T13:28:51.292 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-01-24T13:28:51.293 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/usr/bin/daemon-helper 2024-01-24T13:28:51.327 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-24T13:28:51.397 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-24T13:28:51.397 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-24T13:28:51.397 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-24T13:28:51.431 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-24T13:28:51.505 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-01-24T13:28:51.506 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-24T13:28:51.537 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-24T13:28:51.610 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-01-24T13:28:51.610 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-24T13:28:51.642 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-24T13:28:51.712 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-24T13:28:51.712 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-24T13:28:51.712 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/stdin-killer 2024-01-24T13:28:51.746 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-24T13:28:51.821 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-01-24T13:28:51.821 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/stdin-killer 2024-01-24T13:28:51.854 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-24T13:28:51.924 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-01-24T13:28:51.924 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/usr/bin/stdin-killer 2024-01-24T13:28:51.956 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-24T13:28:52.026 INFO:teuthology.run_tasks:Running task cephadm... 2024-01-24T13:28:52.101 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '4e3182a65800aed6d4aa2f6b86da3cfd72188134', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-01-24T13:28:52.101 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 2024-01-24T13:28:52.102 INFO:tasks.cephadm:Cluster fsid is 83afcf5e-babc-11ee-95b1-87774f69a715 2024-01-24T13:28:52.102 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-01-24T13:28:52.102 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.29', 'mon.b': '172.21.15.163', 'mon.c': '172.21.15.187'} 2024-01-24T13:28:52.102 INFO:tasks.cephadm:First mon is mon.a on smithi029 2024-01-24T13:28:52.102 INFO:tasks.cephadm:First mgr is x 2024-01-24T13:28:52.102 INFO:tasks.cephadm:Normalizing hostnames... 2024-01-24T13:28:52.102 DEBUG:teuthology.orchestra.run.smithi029:> sudo hostname $(hostname -s) 2024-01-24T13:28:52.137 DEBUG:teuthology.orchestra.run.smithi163:> sudo hostname $(hostname -s) 2024-01-24T13:28:52.169 DEBUG:teuthology.orchestra.run.smithi187:> sudo hostname $(hostname -s) 2024-01-24T13:28:52.201 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-01-24T13:28:52.201 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4e3182a65800aed6d4aa2f6b86da3cfd72188134 2024-01-24T13:28:52.342 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '4e3182a65800aed6d4aa2f6b86da3cfd72188134', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/76187/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.2.1-341-g4e3182a6', 'node_name': '172.21.2.7+braggi07', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.2.1-341.g4e3182a6'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2024-01-24 07:52:51.100568', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-vshankar-testing1-reef-2024-01-24-1150', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/8/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-01-24T13:28:52.482 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-vshankar-testing1-reef-2024-01-24-1150, sha1 4e3182a65800aed6d4aa2f6b86da3cfd72188134 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=4e3182a65800aed6d4aa2f6b86da3cfd72188134 2024-01-24T13:28:52.483 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/8/x86_64/flavors/default/cephadm 2024-01-24T13:28:52.484 DEBUG:teuthology.orchestra.run.smithi029:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-24T13:28:53.091 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-rw-r--. 1 ubuntu ubuntu 690687 Jan 24 13:28 /home/ubuntu/cephtest/cephadm 2024-01-24T13:28:53.091 DEBUG:teuthology.orchestra.run.smithi163:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-24T13:28:54.027 INFO:teuthology.orchestra.run.smithi163.stdout:-rw-rw-r--. 1 ubuntu ubuntu 690687 Jan 24 13:28 /home/ubuntu/cephtest/cephadm 2024-01-24T13:28:54.027 DEBUG:teuthology.orchestra.run.smithi187:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-24T13:28:54.695 INFO:teuthology.orchestra.run.smithi187.stdout:-rw-rw-r--. 1 ubuntu ubuntu 690687 Jan 24 13:28 /home/ubuntu/cephtest/cephadm 2024-01-24T13:28:54.696 DEBUG:teuthology.orchestra.run.smithi029:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-24T13:28:54.719 DEBUG:teuthology.orchestra.run.smithi163:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-24T13:28:54.742 DEBUG:teuthology.orchestra.run.smithi187:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-24T13:28:54.780 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 on all hosts... 2024-01-24T13:28:54.780 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 pull 2024-01-24T13:28:54.783 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 pull 2024-01-24T13:28:54.785 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 pull 2024-01-24T13:28:56.186 INFO:teuthology.orchestra.run.smithi163.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134... 2024-01-24T13:28:56.327 INFO:teuthology.orchestra.run.smithi187.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134... 2024-01-24T13:28:58.256 INFO:teuthology.orchestra.run.smithi029.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134... 2024-01-24T13:29:20.799 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-01-24T13:29:20.800 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", 2024-01-24T13:29:20.800 INFO:teuthology.orchestra.run.smithi163.stdout: "image_id": "d650b791ad541e9330061e00985d36774db72ca0aaab13a27fd81b9f0f9eec74", 2024-01-24T13:29:20.800 INFO:teuthology.orchestra.run.smithi163.stdout: "repo_digests": [ 2024-01-24T13:29:20.800 INFO:teuthology.orchestra.run.smithi163.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:22efea2f35520d1212d406fc0d380a7fe7894a099a64f04365d740d4e7cb8285" 2024-01-24T13:29:20.800 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-01-24T13:29:20.800 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-01-24T13:29:25.578 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-01-24T13:29:25.578 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", 2024-01-24T13:29:25.578 INFO:teuthology.orchestra.run.smithi029.stdout: "image_id": "d650b791ad541e9330061e00985d36774db72ca0aaab13a27fd81b9f0f9eec74", 2024-01-24T13:29:25.579 INFO:teuthology.orchestra.run.smithi029.stdout: "repo_digests": [ 2024-01-24T13:29:25.579 INFO:teuthology.orchestra.run.smithi029.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:22efea2f35520d1212d406fc0d380a7fe7894a099a64f04365d740d4e7cb8285" 2024-01-24T13:29:25.579 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-01-24T13:29:25.579 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-01-24T13:29:26.486 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-01-24T13:29:26.487 INFO:teuthology.orchestra.run.smithi187.stdout: "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", 2024-01-24T13:29:26.487 INFO:teuthology.orchestra.run.smithi187.stdout: "image_id": "d650b791ad541e9330061e00985d36774db72ca0aaab13a27fd81b9f0f9eec74", 2024-01-24T13:29:26.487 INFO:teuthology.orchestra.run.smithi187.stdout: "repo_digests": [ 2024-01-24T13:29:26.487 INFO:teuthology.orchestra.run.smithi187.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:22efea2f35520d1212d406fc0d380a7fe7894a099a64f04365d740d4e7cb8285" 2024-01-24T13:29:26.487 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-01-24T13:29:26.487 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-01-24T13:29:26.507 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /etc/ceph 2024-01-24T13:29:26.546 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /etc/ceph 2024-01-24T13:29:26.583 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /etc/ceph 2024-01-24T13:29:26.613 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 777 /etc/ceph 2024-01-24T13:29:26.648 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 777 /etc/ceph 2024-01-24T13:29:26.679 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 777 /etc/ceph 2024-01-24T13:29:26.706 INFO:tasks.cephadm:Writing seed config... 2024-01-24T13:29:26.707 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-01-24T13:29:26.707 INFO:tasks.cephadm: override: [client] debug client = 20 2024-01-24T13:29:26.707 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-01-24T13:29:26.707 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-01-24T13:29:26.707 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-01-24T13:29:26.707 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-01-24T13:29:26.708 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-01-24T13:29:26.708 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-01-24T13:29:26.708 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-01-24T13:29:26.708 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-01-24T13:29:26.708 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-01-24T13:29:26.708 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-01-24T13:29:26.708 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-01-24T13:29:26.708 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-01-24T13:29:26.708 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-01-24T13:29:26.708 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-01-24T13:29:26.708 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-01-24T13:29:26.708 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-01-24T13:29:26.708 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-01-24T13:29:26.708 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-01-24T13:29:26.708 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-01-24T13:29:26.709 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-01-24T13:29:26.709 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-01-24T13:29:26.709 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-01-24T13:29:26.709 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-01-24T13:29:26.709 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-01-24T13:29:26.709 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-01-24T13:29:26.709 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-01-24T13:29:26.709 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-01-24T13:29:26.709 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-01-24T13:29:26.709 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-01-24T13:29:26.709 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-01-24T13:29:26.709 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-01-24T13:29:26.709 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-01-24T13:29:26.709 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-01-24T13:29:26.709 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-01-24T13:29:26.710 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-01-24T13:29:26.710 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-01-24T13:29:26.710 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-01-24T13:29:26.710 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-01-24T13:29:26.710 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-01-24T13:29:26.710 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-01-24T13:29:26.710 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-01-24T13:29:26.711 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-24T13:29:26.711 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-01-24T13:29:26.732 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 83afcf5e-babc-11ee-95b1-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-01-24T13:29:26.732 DEBUG:teuthology.orchestra.run.smithi029:mon.a> sudo journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mon.a.service 2024-01-24T13:29:26.775 DEBUG:teuthology.orchestra.run.smithi029:mgr.x> sudo journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mgr.x.service 2024-01-24T13:29:26.810 INFO:journalctl@ceph.mon.a.smithi029.stdout:-- Logs begin at Wed 2024-01-24 13:13:42 UTC. -- 2024-01-24T13:29:26.811 INFO:tasks.cephadm:Bootstrapping... 2024-01-24T13:29:26.811 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 -v bootstrap --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.29 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-01-24T13:29:26.843 INFO:journalctl@ceph.mgr.x.smithi029.stdout:-- Logs begin at Wed 2024-01-24 13:13:42 UTC. -- 2024-01-24T13:29:26.999 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-01-24T13:29:26.999 INFO:teuthology.orchestra.run.smithi029.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134', '-v', 'bootstrap', '--fsid', '83afcf5e-babc-11ee-95b1-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.29', '--skip-admin-label'] 2024-01-24T13:29:27.038 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stdout 4.6.1 2024-01-24T13:29:27.038 INFO:teuthology.orchestra.run.smithi029.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-01-24T13:29:27.038 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying podman|docker is present... 2024-01-24T13:29:27.072 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stdout 4.6.1 2024-01-24T13:29:27.072 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying lvm2 is present... 2024-01-24T13:29:27.072 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying time synchronization is in place... 2024-01-24T13:29:27.080 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-01-24T13:29:27.080 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-01-24T13:29:27.086 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-01-24T13:29:27.086 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-01-24T13:29:27.093 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout enabled 2024-01-24T13:29:27.100 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout active 2024-01-24T13:29:27.100 INFO:teuthology.orchestra.run.smithi029.stdout:Unit chronyd.service is enabled and running 2024-01-24T13:29:27.100 INFO:teuthology.orchestra.run.smithi029.stdout:Repeating the final host check... 2024-01-24T13:29:27.137 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stdout 4.6.1 2024-01-24T13:29:27.137 INFO:teuthology.orchestra.run.smithi029.stdout:podman (/bin/podman) version 4.6.1 is present 2024-01-24T13:29:27.137 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl is present 2024-01-24T13:29:27.137 INFO:teuthology.orchestra.run.smithi029.stdout:lvcreate is present 2024-01-24T13:29:27.144 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-01-24T13:29:27.144 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-01-24T13:29:27.150 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-01-24T13:29:27.150 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-01-24T13:29:27.157 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout enabled 2024-01-24T13:29:27.164 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout active 2024-01-24T13:29:27.164 INFO:teuthology.orchestra.run.smithi029.stdout:Unit chronyd.service is enabled and running 2024-01-24T13:29:27.164 INFO:teuthology.orchestra.run.smithi029.stdout:Host looks OK 2024-01-24T13:29:27.164 INFO:teuthology.orchestra.run.smithi029.stdout:Cluster fsid: 83afcf5e-babc-11ee-95b1-87774f69a715 2024-01-24T13:29:27.164 INFO:teuthology.orchestra.run.smithi029.stdout:Acquiring lock 139746865494504 on /run/cephadm/83afcf5e-babc-11ee-95b1-87774f69a715.lock 2024-01-24T13:29:27.164 INFO:teuthology.orchestra.run.smithi029.stdout:Lock 139746865494504 acquired on /run/cephadm/83afcf5e-babc-11ee-95b1-87774f69a715.lock 2024-01-24T13:29:27.164 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying IP 172.21.15.29 port 3300 ... 2024-01-24T13:29:27.165 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying IP 172.21.15.29 port 6789 ... 2024-01-24T13:29:27.165 INFO:teuthology.orchestra.run.smithi029.stdout:Base mon IP(s) is [172.21.15.29:3300, 172.21.15.29:6789], mon addrv is [v2:172.21.15.29:3300,v1:172.21.15.29:6789] 2024-01-24T13:29:27.169 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp metric 100 2024-01-24T13:29:27.169 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.29 metric 100 2024-01-24T13:29:27.173 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-01-24T13:29:27.173 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-01-24T13:29:27.173 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1784sec hoplimit 64 pref medium 2024-01-24T13:29:27.176 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-01-24T13:29:27.176 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-01-24T13:29:27.176 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-01-24T13:29:27.176 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout 3: ens1f0: mtu 1500 state UP qlen 1000 2024-01-24T13:29:27.176 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:125a/64 scope link 2024-01-24T13:29:27.176 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-01-24T13:29:27.177 INFO:teuthology.orchestra.run.smithi029.stdout:Mon IP `172.21.15.29` is in CIDR network `172.21.0.0/20` 2024-01-24T13:29:27.177 INFO:teuthology.orchestra.run.smithi029.stdout:Mon IP `172.21.15.29` is in CIDR network `172.21.0.0/20` 2024-01-24T13:29:27.177 INFO:teuthology.orchestra.run.smithi029.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-01-24T13:29:27.177 INFO:teuthology.orchestra.run.smithi029.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-01-24T13:29:27.179 INFO:teuthology.orchestra.run.smithi029.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134... 2024-01-24T13:29:27.542 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stdout d650b791ad541e9330061e00985d36774db72ca0aaab13a27fd81b9f0f9eec74 2024-01-24T13:29:27.542 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134... 2024-01-24T13:29:27.543 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Getting image source signatures 2024-01-24T13:29:27.543 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Copying blob sha256:5bd92459d7edc3325b5efa151665fd479e83585b213707287afb2af065f59941 2024-01-24T13:29:27.543 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Copying blob sha256:ea3366b8a90265ae3f6d3736cb0de4b52da4ffec919b46c6cb5bc5be1372efe8 2024-01-24T13:29:27.543 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Copying config sha256:d650b791ad541e9330061e00985d36774db72ca0aaab13a27fd81b9f0f9eec74 2024-01-24T13:29:27.543 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Writing manifest to image destination 2024-01-24T13:29:28.256 INFO:teuthology.orchestra.run.smithi029.stdout:ceph: stdout ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable) 2024-01-24T13:29:28.256 INFO:teuthology.orchestra.run.smithi029.stdout:Ceph version: ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable) 2024-01-24T13:29:28.257 INFO:teuthology.orchestra.run.smithi029.stdout:Extracting ceph user uid/gid from container image... 2024-01-24T13:29:28.893 INFO:teuthology.orchestra.run.smithi029.stdout:stat: stdout 167 167 2024-01-24T13:29:28.894 INFO:teuthology.orchestra.run.smithi029.stdout:Creating initial keys... 2024-01-24T13:29:29.510 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph-authtool: stdout AQC5ELFlRyuMCRAA6uqDW/X2mhvSFED9VoxTEQ== 2024-01-24T13:29:30.110 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph-authtool: stdout AQC5ELFlJf42LRAACC4yMW7VQtVCJUmrL6ZYOg== 2024-01-24T13:29:30.755 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph-authtool: stdout AQC6ELFlhCLxFxAAKiOPlRYd9NudfiuisTaoaA== 2024-01-24T13:29:30.756 INFO:teuthology.orchestra.run.smithi029.stdout:Creating initial monmap... 2024-01-24T13:29:31.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-24T13:29:31.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-01-24T13:29:31.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 83afcf5e-babc-11ee-95b1-87774f69a715 2024-01-24T13:29:31.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-24T13:29:31.383 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool for a [v2:172.21.15.29:3300,v1:172.21.15.29:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-24T13:29:31.383 INFO:teuthology.orchestra.run.smithi029.stdout:setting min_mon_release = pacific 2024-01-24T13:29:31.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: set fsid to 83afcf5e-babc-11ee-95b1-87774f69a715 2024-01-24T13:29:31.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-24T13:29:31.383 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:29:31.384 INFO:teuthology.orchestra.run.smithi029.stdout:Creating mon... 2024-01-24T13:29:32.030 INFO:teuthology.orchestra.run.smithi029.stdout:create mon.a on 2024-01-24T13:29:32.179 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-01-24T13:29:32.278 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-01-24T13:29:32.412 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-83afcf5e-babc-11ee-95b1-87774f69a715.target → /etc/systemd/system/ceph-83afcf5e-babc-11ee-95b1-87774f69a715.target. 2024-01-24T13:29:32.412 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-83afcf5e-babc-11ee-95b1-87774f69a715.target → /etc/systemd/system/ceph-83afcf5e-babc-11ee-95b1-87774f69a715.target. 2024-01-24T13:29:32.610 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mon.a 2024-01-24T13:29:32.610 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Failed to reset failed state of unit ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mon.a.service: Unit ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mon.a.service not loaded. 2024-01-24T13:29:32.711 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-83afcf5e-babc-11ee-95b1-87774f69a715.target.wants/ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mon.a.service → /etc/systemd/system/ceph-83afcf5e-babc-11ee-95b1-87774f69a715@.service. 2024-01-24T13:29:33.152 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-01-24T13:29:33.152 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout disabled 2024-01-24T13:29:33.161 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-01-24T13:29:33.161 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-01-24T13:29:33.161 INFO:teuthology.orchestra.run.smithi029.stdout:firewalld.service is not enabled 2024-01-24T13:29:33.161 INFO:teuthology.orchestra.run.smithi029.stdout:Not possible to enable service . firewalld.service is not available 2024-01-24T13:29:33.162 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mon to start... 2024-01-24T13:29:33.162 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mon... 2024-01-24T13:29:33.313 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:33 smithi029 systemd[1]: Started Ceph mon.a for 83afcf5e-babc-11ee-95b1-87774f69a715. 2024-01-24T13:29:33.314 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:33 smithi029 ceph-mon[40166]: mkfs 83afcf5e-babc-11ee-95b1-87774f69a715 2024-01-24T13:29:33.314 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:33 smithi029 ceph-mon[40166]: mon.a is new leader, mons a in quorum (ranks 0) 2024-01-24T13:29:33.962 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout cluster: 2024-01-24T13:29:33.966 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout id: 83afcf5e-babc-11ee-95b1-87774f69a715 2024-01-24T13:29:33.966 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-01-24T13:29:33.966 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-01-24T13:29:33.966 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout services: 2024-01-24T13:29:33.966 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.488692s) 2024-01-24T13:29:33.966 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-01-24T13:29:33.966 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-01-24T13:29:33.966 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-01-24T13:29:33.966 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout data: 2024-01-24T13:29:33.966 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-01-24T13:29:33.966 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-01-24T13:29:33.967 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-01-24T13:29:33.967 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout pgs: 2024-01-24T13:29:33.967 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-01-24T13:29:33.967 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.579+0000 7f2159be0700 1 Processor -- start 2024-01-24T13:29:33.967 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.579+0000 7f2159be0700 1 -- start start 2024-01-24T13:29:33.967 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.579+0000 7f2159be0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2154106fd0 0x7f21541093c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:29:33.967 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.579+0000 7f2159be0700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2154079dd0 con 0x7f2154106fd0 2024-01-24T13:29:33.967 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.580+0000 7f21537fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2154106fd0 0x7f21541093c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:29:33.967 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.580+0000 7f21537fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2154106fd0 0x7f21541093c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35760/0 (socket says 172.21.15.29:35760) 2024-01-24T13:29:33.967 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.580+0000 7f21537fe700 1 -- 172.21.15.29:0/2625187215 learned_addr learned my addr 172.21.15.29:0/2625187215 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:29:33.967 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.580+0000 7f21537fe700 1 -- 172.21.15.29:0/2625187215 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2154079f10 con 0x7f2154106fd0 2024-01-24T13:29:33.967 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.580+0000 7f21537fe700 1 --2- 172.21.15.29:0/2625187215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2154106fd0 0x7f21541093c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f21440098a0 tx=0x7f2144009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b902fd7782943602 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:29:33.967 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.580+0000 7f21527fc700 1 -- 172.21.15.29:0/2625187215 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2144004160 con 0x7f2154106fd0 2024-01-24T13:29:33.967 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.580+0000 7f21527fc700 1 -- 172.21.15.29:0/2625187215 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21440042c0 con 0x7f2154106fd0 2024-01-24T13:29:33.968 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.580+0000 7f21527fc700 1 -- 172.21.15.29:0/2625187215 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f21440045a0 con 0x7f2154106fd0 2024-01-24T13:29:33.968 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.581+0000 7f2159be0700 1 -- 172.21.15.29:0/2625187215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2154106fd0 msgr2=0x7f21541093c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:29:33.968 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.581+0000 7f2159be0700 1 --2- 172.21.15.29:0/2625187215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2154106fd0 0x7f21541093c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f21440098a0 tx=0x7f2144009bb0 comp rx=0 tx=0).stop 2024-01-24T13:29:33.968 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.581+0000 7f2159be0700 1 -- 172.21.15.29:0/2625187215 shutdown_connections 2024-01-24T13:29:33.968 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.581+0000 7f2159be0700 1 --2- 172.21.15.29:0/2625187215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2154106fd0 0x7f21541093c0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:29:33.968 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.581+0000 7f2159be0700 1 -- 172.21.15.29:0/2625187215 >> 172.21.15.29:0/2625187215 conn(0x7f2154100980 msgr2=0x7f2154102da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:29:33.968 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.581+0000 7f2159be0700 1 -- 172.21.15.29:0/2625187215 shutdown_connections 2024-01-24T13:29:33.968 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.581+0000 7f2159be0700 1 -- 172.21.15.29:0/2625187215 wait complete. 2024-01-24T13:29:33.968 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.582+0000 7f2159be0700 1 Processor -- start 2024-01-24T13:29:33.968 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.582+0000 7f2159be0700 1 -- start start 2024-01-24T13:29:33.968 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.583+0000 7f2159be0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2154106fd0 0x7f21541998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:29:33.968 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.583+0000 7f2159be0700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2154199df0 con 0x7f2154106fd0 2024-01-24T13:29:33.971 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.583+0000 7f21537fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2154106fd0 0x7f21541998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:29:33.971 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.583+0000 7f21537fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2154106fd0 0x7f21541998b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35766/0 (socket says 172.21.15.29:35766) 2024-01-24T13:29:33.971 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.583+0000 7f21537fe700 1 -- 172.21.15.29:0/3986381947 learned_addr learned my addr 172.21.15.29:0/3986381947 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:29:33.971 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.583+0000 7f21537fe700 1 -- 172.21.15.29:0/3986381947 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2144009550 con 0x7f2154106fd0 2024-01-24T13:29:33.971 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.583+0000 7f21537fe700 1 --2- 172.21.15.29:0/3986381947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2154106fd0 0x7f21541998b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2154079210 tx=0x7f2144004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:29:33.971 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.583+0000 7f2150ff9700 1 -- 172.21.15.29:0/3986381947 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2144011630 con 0x7f2154106fd0 2024-01-24T13:29:33.971 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.583+0000 7f2150ff9700 1 -- 172.21.15.29:0/3986381947 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2144011790 con 0x7f2154106fd0 2024-01-24T13:29:33.972 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.583+0000 7f2159be0700 1 -- 172.21.15.29:0/3986381947 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2154199ff0 con 0x7f2154106fd0 2024-01-24T13:29:33.972 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.583+0000 7f2150ff9700 1 -- 172.21.15.29:0/3986381947 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2144011a30 con 0x7f2154106fd0 2024-01-24T13:29:33.972 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.584+0000 7f2159be0700 1 -- 172.21.15.29:0/3986381947 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f215419a490 con 0x7f2154106fd0 2024-01-24T13:29:33.972 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.584+0000 7f2150ff9700 1 -- 172.21.15.29:0/3986381947 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f2144022070 con 0x7f2154106fd0 2024-01-24T13:29:33.972 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.584+0000 7f2150ff9700 1 -- 172.21.15.29:0/3986381947 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f214401c790 con 0x7f2154106fd0 2024-01-24T13:29:33.972 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.585+0000 7f213a7fc700 1 -- 172.21.15.29:0/3986381947 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21340052f0 con 0x7f2154106fd0 2024-01-24T13:29:33.972 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.587+0000 7f2150ff9700 1 -- 172.21.15.29:0/3986381947 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f214401bcf0 con 0x7f2154106fd0 2024-01-24T13:29:33.972 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.662+0000 7f213a7fc700 1 -- 172.21.15.29:0/3986381947 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f2134005c90 con 0x7f2154106fd0 2024-01-24T13:29:33.972 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.663+0000 7f2150ff9700 1 -- 172.21.15.29:0/3986381947 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7f214401b450 con 0x7f2154106fd0 2024-01-24T13:29:33.972 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.664+0000 7f213a7fc700 1 -- 172.21.15.29:0/3986381947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2154106fd0 msgr2=0x7f21541998b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:29:33.972 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.664+0000 7f213a7fc700 1 --2- 172.21.15.29:0/3986381947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2154106fd0 0x7f21541998b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2154079210 tx=0x7f2144004750 comp rx=0 tx=0).stop 2024-01-24T13:29:33.972 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.664+0000 7f213a7fc700 1 -- 172.21.15.29:0/3986381947 shutdown_connections 2024-01-24T13:29:33.972 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.664+0000 7f213a7fc700 1 --2- 172.21.15.29:0/3986381947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2154106fd0 0x7f21541998b0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:29:33.972 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.664+0000 7f213a7fc700 1 -- 172.21.15.29:0/3986381947 >> 172.21.15.29:0/3986381947 conn(0x7f2154100980 msgr2=0x7f2154102240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:29:33.973 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.664+0000 7f213a7fc700 1 -- 172.21.15.29:0/3986381947 shutdown_connections 2024-01-24T13:29:33.973 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:33.664+0000 7f213a7fc700 1 -- 172.21.15.29:0/3986381947 wait complete. 2024-01-24T13:29:33.973 INFO:teuthology.orchestra.run.smithi029.stdout:mon is available 2024-01-24T13:29:33.973 INFO:teuthology.orchestra.run.smithi029.stdout:Assimilating anything we can from ceph.conf... 2024-01-24T13:29:34.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:34 smithi029 ceph-mon[40166]: mon.a is new leader, mons a in quorum (ranks 0) 2024-01-24T13:29:34.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:34 smithi029 ceph-mon[40166]: monmap e1: 1 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-01-24T13:29:34.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:34 smithi029 ceph-mon[40166]: fsmap 2024-01-24T13:29:34.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:34 smithi029 ceph-mon[40166]: osdmap e1: 0 total, 0 up, 0 in 2024-01-24T13:29:34.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:34 smithi029 ceph-mon[40166]: mgrmap e1: no daemons active 2024-01-24T13:29:34.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:34 smithi029 ceph-mon[40166]: from='client.? 172.21.15.29:0/3986381947' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-01-24T13:29:35.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:35 smithi029 ceph-mon[40166]: from='client.? 172.21.15.29:0/1640043219' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-24T13:29:35.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:35 smithi029 ceph-mon[40166]: from='client.? 172.21.15.29:0/1640043219' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-01-24T13:29:37.454 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-01-24T13:29:37.454 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [global] 2024-01-24T13:29:37.454 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout fsid = 83afcf5e-babc-11ee-95b1-87774f69a715 2024-01-24T13:29:37.454 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.29:3300,v1:172.21.15.29:6789] 2024-01-24T13:29:37.454 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-01-24T13:29:37.454 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-01-24T13:29:37.454 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-01-24T13:29:37.454 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-01-24T13:29:37.454 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-01-24T13:29:37.454 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [mgr] 2024-01-24T13:29:37.455 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-01-24T13:29:37.455 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-01-24T13:29:37.455 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [osd] 2024-01-24T13:29:37.455 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-01-24T13:29:37.455 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-01-24T13:29:37.455 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-01-24T13:29:37.455 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-01-24T13:29:37.455 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-01-24T13:29:37.455 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.448+0000 7f2fc9818700 1 Processor -- start 2024-01-24T13:29:37.455 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.448+0000 7f2fc9818700 1 -- start start 2024-01-24T13:29:37.455 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.448+0000 7f2fc9818700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fc4106f60 0x7f2fc4109350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:29:37.455 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.448+0000 7f2fc9818700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc4079e50 con 0x7f2fc4106f60 2024-01-24T13:29:37.456 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.448+0000 7f2fc2ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fc4106f60 0x7f2fc4109350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:29:37.456 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.448+0000 7f2fc2ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fc4106f60 0x7f2fc4109350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35774/0 (socket says 172.21.15.29:35774) 2024-01-24T13:29:37.456 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.448+0000 7f2fc2ffd700 1 -- 172.21.15.29:0/1001002726 learned_addr learned my addr 172.21.15.29:0/1001002726 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:29:37.456 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.448+0000 7f2fc2ffd700 1 -- 172.21.15.29:0/1001002726 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc4109890 con 0x7f2fc4106f60 2024-01-24T13:29:37.456 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.449+0000 7f2fc2ffd700 1 --2- 172.21.15.29:0/1001002726 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fc4106f60 0x7f2fc4109350 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f2fb40098a0 tx=0x7f2fb4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53c7631cbf5bd9eb server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:29:37.456 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.449+0000 7f2fc1ffb700 1 -- 172.21.15.29:0/1001002726 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2fb4004160 con 0x7f2fc4106f60 2024-01-24T13:29:37.456 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.449+0000 7f2fc1ffb700 1 -- 172.21.15.29:0/1001002726 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb40042c0 con 0x7f2fc4106f60 2024-01-24T13:29:37.456 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.449+0000 7f2fc1ffb700 1 -- 172.21.15.29:0/1001002726 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2fb40045a0 con 0x7f2fc4106f60 2024-01-24T13:29:37.456 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.449+0000 7f2fc9818700 1 -- 172.21.15.29:0/1001002726 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fc4106f60 msgr2=0x7f2fc4109350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:29:37.456 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.449+0000 7f2fc9818700 1 --2- 172.21.15.29:0/1001002726 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fc4106f60 0x7f2fc4109350 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f2fb40098a0 tx=0x7f2fb4009bb0 comp rx=0 tx=0).stop 2024-01-24T13:29:37.456 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.449+0000 7f2fc9818700 1 -- 172.21.15.29:0/1001002726 shutdown_connections 2024-01-24T13:29:37.456 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.449+0000 7f2fc9818700 1 --2- 172.21.15.29:0/1001002726 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fc4106f60 0x7f2fc4109350 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:29:37.456 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.449+0000 7f2fc9818700 1 -- 172.21.15.29:0/1001002726 >> 172.21.15.29:0/1001002726 conn(0x7f2fc41009b0 msgr2=0x7f2fc4102dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:29:37.456 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.449+0000 7f2fc9818700 1 -- 172.21.15.29:0/1001002726 shutdown_connections 2024-01-24T13:29:37.457 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.449+0000 7f2fc9818700 1 -- 172.21.15.29:0/1001002726 wait complete. 2024-01-24T13:29:37.457 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.450+0000 7f2fc9818700 1 Processor -- start 2024-01-24T13:29:37.457 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.450+0000 7f2fc9818700 1 -- start start 2024-01-24T13:29:37.457 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.450+0000 7f2fc9818700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fc4106f60 0x7f2fc41998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:29:37.459 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.450+0000 7f2fc9818700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc4199e10 con 0x7f2fc4106f60 2024-01-24T13:29:37.460 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.451+0000 7f2fc2ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fc4106f60 0x7f2fc41998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:29:37.460 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.451+0000 7f2fc2ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fc4106f60 0x7f2fc41998d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35786/0 (socket says 172.21.15.29:35786) 2024-01-24T13:29:37.460 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.451+0000 7f2fc2ffd700 1 -- 172.21.15.29:0/1640043219 learned_addr learned my addr 172.21.15.29:0/1640043219 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:29:37.460 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.451+0000 7f2fc2ffd700 1 -- 172.21.15.29:0/1640043219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb4009550 con 0x7f2fc4106f60 2024-01-24T13:29:37.460 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.451+0000 7f2fc2ffd700 1 --2- 172.21.15.29:0/1640043219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fc4106f60 0x7f2fc41998d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4079290 tx=0x7f2fb4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:29:37.460 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.451+0000 7f2fc8816700 1 -- 172.21.15.29:0/1640043219 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2fb4011630 con 0x7f2fc4106f60 2024-01-24T13:29:37.460 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.451+0000 7f2fc8816700 1 -- 172.21.15.29:0/1640043219 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb4011790 con 0x7f2fc4106f60 2024-01-24T13:29:37.460 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.451+0000 7f2fc9818700 1 -- 172.21.15.29:0/1640043219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fc419a010 con 0x7f2fc4106f60 2024-01-24T13:29:37.460 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.451+0000 7f2fc8816700 1 -- 172.21.15.29:0/1640043219 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2fb4011a30 con 0x7f2fc4106f60 2024-01-24T13:29:37.460 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.451+0000 7f2fc9818700 1 -- 172.21.15.29:0/1640043219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fc419a4b0 con 0x7f2fc4106f60 2024-01-24T13:29:37.460 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.451+0000 7f2fc8816700 1 -- 172.21.15.29:0/1640043219 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f2fb4022070 con 0x7f2fc4106f60 2024-01-24T13:29:37.460 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.451+0000 7f2fc8816700 1 -- 172.21.15.29:0/1640043219 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2fb401c450 con 0x7f2fc4106f60 2024-01-24T13:29:37.461 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.452+0000 7f2fc9818700 1 -- 172.21.15.29:0/1640043219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fc41937b0 con 0x7f2fc4106f60 2024-01-24T13:29:37.461 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.455+0000 7f2fc8816700 1 -- 172.21.15.29:0/1640043219 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f2fb401b450 con 0x7f2fc4106f60 2024-01-24T13:29:37.461 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.529+0000 7f2fc9818700 1 -- 172.21.15.29:0/1640043219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f2fc402cf20 con 0x7f2fc4106f60 2024-01-24T13:29:37.461 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.532+0000 7f2fc8816700 1 -- 172.21.15.29:0/1640043219 <== mon.0 v2:172.21.15.29:3300/0 7 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2fb401b5f0 con 0x7f2fc4106f60 2024-01-24T13:29:37.461 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.532+0000 7f2fc8816700 1 -- 172.21.15.29:0/1640043219 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7f2fb4024630 con 0x7f2fc4106f60 2024-01-24T13:29:37.461 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.533+0000 7f2fc9818700 1 -- 172.21.15.29:0/1640043219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fc4106f60 msgr2=0x7f2fc41998d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:29:37.461 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.533+0000 7f2fc9818700 1 --2- 172.21.15.29:0/1640043219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fc4106f60 0x7f2fc41998d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4079290 tx=0x7f2fb4004750 comp rx=0 tx=0).stop 2024-01-24T13:29:37.461 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.533+0000 7f2fc9818700 1 -- 172.21.15.29:0/1640043219 shutdown_connections 2024-01-24T13:29:37.461 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.533+0000 7f2fc9818700 1 --2- 172.21.15.29:0/1640043219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fc4106f60 0x7f2fc41998d0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:29:37.461 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.533+0000 7f2fc9818700 1 -- 172.21.15.29:0/1640043219 >> 172.21.15.29:0/1640043219 conn(0x7f2fc41009b0 msgr2=0x7f2fc4102290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:29:37.461 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.534+0000 7f2fc9818700 1 -- 172.21.15.29:0/1640043219 shutdown_connections 2024-01-24T13:29:37.461 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:35.534+0000 7f2fc9818700 1 -- 172.21.15.29:0/1640043219 wait complete. 2024-01-24T13:29:37.461 INFO:teuthology.orchestra.run.smithi029.stdout:Generating new minimal ceph.conf... 2024-01-24T13:29:38.807 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.340+0000 7fda91418700 1 Processor -- start 2024-01-24T13:29:38.808 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.340+0000 7fda91418700 1 -- start start 2024-01-24T13:29:38.808 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.340+0000 7fda91418700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda8c106f10 0x7fda8c109300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:29:38.808 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.340+0000 7fda91418700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8c079e90 con 0x7fda8c106f10 2024-01-24T13:29:38.808 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.341+0000 7fda8affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda8c106f10 0x7fda8c109300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:29:38.808 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.341+0000 7fda8affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda8c106f10 0x7fda8c109300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35794/0 (socket says 172.21.15.29:35794) 2024-01-24T13:29:38.808 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.341+0000 7fda8affd700 1 -- 172.21.15.29:0/3516532429 learned_addr learned my addr 172.21.15.29:0/3516532429 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:29:38.808 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.341+0000 7fda8affd700 1 -- 172.21.15.29:0/3516532429 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda8c109840 con 0x7fda8c106f10 2024-01-24T13:29:38.808 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.341+0000 7fda8affd700 1 --2- 172.21.15.29:0/3516532429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda8c106f10 0x7fda8c109300 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fda7c0098a0 tx=0x7fda7c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eddd8d5d21653a02 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:29:38.808 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.341+0000 7fda89ffb700 1 -- 172.21.15.29:0/3516532429 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fda7c004160 con 0x7fda8c106f10 2024-01-24T13:29:38.808 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.341+0000 7fda89ffb700 1 -- 172.21.15.29:0/3516532429 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda7c0042c0 con 0x7fda8c106f10 2024-01-24T13:29:38.808 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.341+0000 7fda89ffb700 1 -- 172.21.15.29:0/3516532429 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fda7c011a00 con 0x7fda8c106f10 2024-01-24T13:29:38.809 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.342+0000 7fda91418700 1 -- 172.21.15.29:0/3516532429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda8c106f10 msgr2=0x7fda8c109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:29:38.809 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.342+0000 7fda91418700 1 --2- 172.21.15.29:0/3516532429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda8c106f10 0x7fda8c109300 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fda7c0098a0 tx=0x7fda7c009bb0 comp rx=0 tx=0).stop 2024-01-24T13:29:38.809 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.342+0000 7fda91418700 1 -- 172.21.15.29:0/3516532429 shutdown_connections 2024-01-24T13:29:38.809 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.342+0000 7fda91418700 1 --2- 172.21.15.29:0/3516532429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda8c106f10 0x7fda8c109300 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:29:38.809 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.342+0000 7fda91418700 1 -- 172.21.15.29:0/3516532429 >> 172.21.15.29:0/3516532429 conn(0x7fda8c100980 msgr2=0x7fda8c102da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:29:38.809 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.342+0000 7fda91418700 1 -- 172.21.15.29:0/3516532429 shutdown_connections 2024-01-24T13:29:38.809 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.342+0000 7fda91418700 1 -- 172.21.15.29:0/3516532429 wait complete. 2024-01-24T13:29:38.809 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.343+0000 7fda91418700 1 Processor -- start 2024-01-24T13:29:38.809 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.343+0000 7fda91418700 1 -- start start 2024-01-24T13:29:38.809 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.343+0000 7fda91418700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda8c106f10 0x7fda8c19a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:29:38.809 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.343+0000 7fda91418700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8c19a5c0 con 0x7fda8c106f10 2024-01-24T13:29:38.809 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.343+0000 7fda8affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda8c106f10 0x7fda8c19a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:29:38.809 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.343+0000 7fda8affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda8c106f10 0x7fda8c19a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35796/0 (socket says 172.21.15.29:35796) 2024-01-24T13:29:38.809 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.343+0000 7fda8affd700 1 -- 172.21.15.29:0/1625670556 learned_addr learned my addr 172.21.15.29:0/1625670556 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:29:38.810 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.344+0000 7fda8affd700 1 -- 172.21.15.29:0/1625670556 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda7c009550 con 0x7fda8c106f10 2024-01-24T13:29:38.810 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.344+0000 7fda8affd700 1 --2- 172.21.15.29:0/1625670556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda8c106f10 0x7fda8c19a080 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fda7c009fb0 tx=0x7fda7c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:29:38.810 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.344+0000 7fda73fff700 1 -- 172.21.15.29:0/1625670556 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fda7c003930 con 0x7fda8c106f10 2024-01-24T13:29:38.810 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.344+0000 7fda73fff700 1 -- 172.21.15.29:0/1625670556 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda7c003a90 con 0x7fda8c106f10 2024-01-24T13:29:38.810 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.344+0000 7fda73fff700 1 -- 172.21.15.29:0/1625670556 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fda7c019610 con 0x7fda8c106f10 2024-01-24T13:29:38.810 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.344+0000 7fda91418700 1 -- 172.21.15.29:0/1625670556 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda8c19a7c0 con 0x7fda8c106f10 2024-01-24T13:29:38.810 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.344+0000 7fda91418700 1 -- 172.21.15.29:0/1625670556 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda8c19ac60 con 0x7fda8c106f10 2024-01-24T13:29:38.810 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.344+0000 7fda73fff700 1 -- 172.21.15.29:0/1625670556 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fda7c019bd0 con 0x7fda8c106f10 2024-01-24T13:29:38.810 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.344+0000 7fda73fff700 1 -- 172.21.15.29:0/1625670556 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fda7c0243f0 con 0x7fda8c106f10 2024-01-24T13:29:38.810 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.345+0000 7fda91418700 1 -- 172.21.15.29:0/1625670556 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda8c062370 con 0x7fda8c106f10 2024-01-24T13:29:38.810 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.348+0000 7fda73fff700 1 -- 172.21.15.29:0/1625670556 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fda7c003c00 con 0x7fda8c106f10 2024-01-24T13:29:38.810 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.421+0000 7fda91418700 1 -- 172.21.15.29:0/1625670556 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fda8c104f30 con 0x7fda8c106f10 2024-01-24T13:29:38.810 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.421+0000 7fda73fff700 1 -- 172.21.15.29:0/1625670556 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7fda7c01a9d0 con 0x7fda8c106f10 2024-01-24T13:29:38.810 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.422+0000 7fda91418700 1 -- 172.21.15.29:0/1625670556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda8c106f10 msgr2=0x7fda8c19a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:29:38.810 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.422+0000 7fda91418700 1 --2- 172.21.15.29:0/1625670556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda8c106f10 0x7fda8c19a080 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fda7c009fb0 tx=0x7fda7c011ad0 comp rx=0 tx=0).stop 2024-01-24T13:29:38.811 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.422+0000 7fda91418700 1 -- 172.21.15.29:0/1625670556 shutdown_connections 2024-01-24T13:29:38.811 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.422+0000 7fda91418700 1 --2- 172.21.15.29:0/1625670556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda8c106f10 0x7fda8c19a080 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:29:38.811 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.422+0000 7fda91418700 1 -- 172.21.15.29:0/1625670556 >> 172.21.15.29:0/1625670556 conn(0x7fda8c100980 msgr2=0x7fda8c1016d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:29:38.811 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.422+0000 7fda91418700 1 -- 172.21.15.29:0/1625670556 shutdown_connections 2024-01-24T13:29:38.811 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:38.422+0000 7fda91418700 1 -- 172.21.15.29:0/1625670556 wait complete. 2024-01-24T13:29:38.811 INFO:teuthology.orchestra.run.smithi029.stdout:Restarting the monitor... 2024-01-24T13:29:39.033 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:38 smithi029 systemd[1]: Stopping Ceph mon.a for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:29:39.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mon-a[40142]: 2024-01-24T13:29:39.032+0000 7fbc3bae9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T13:29:39.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mon-a[40142]: 2024-01-24T13:29:39.032+0000 7fbc3bae9700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-01-24T13:29:39.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 bash[40492]: ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mon-a 2024-01-24T13:29:39.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 systemd[1]: ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mon.a.service: Succeeded. 2024-01-24T13:29:39.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 systemd[1]: Stopped Ceph mon.a for 83afcf5e-babc-11ee-95b1-87774f69a715. 2024-01-24T13:29:39.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 systemd[1]: Starting Ceph mon.a for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:29:39.978 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 bash[40613]: eabe1f92e9fcce2443b678af641dc4eb93a382e56b7f698020aa6855620173c1 2024-01-24T13:29:39.978 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: set uid:gid to 167:167 (ceph:ceph) 2024-01-24T13:29:39.980 INFO:teuthology.orchestra.run.smithi029.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-01-24T13:29:40.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable), process ceph-mon, pid 7 2024-01-24T13:29:40.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: pidfile_write: ignore empty --pid-file 2024-01-24T13:29:40.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 systemd[1]: Started Ceph mon.a for 83afcf5e-babc-11ee-95b1-87774f69a715. 2024-01-24T13:29:40.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: load: jerasure load: lrc 2024-01-24T13:29:40.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: RocksDB version: 7.9.2 2024-01-24T13:29:40.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Git sha 0 2024-01-24T13:29:40.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Compile date 2024-01-24 06:39:28 2024-01-24T13:29:40.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: DB SUMMARY 2024-01-24T13:29:40.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: DB Session ID: BWI8OTCDEHQ9RF7JWUG8 2024-01-24T13:29:40.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: CURRENT file: CURRENT 2024-01-24T13:29:40.351 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: IDENTITY file: IDENTITY 2024-01-24T13:29:40.351 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-01-24T13:29:40.351 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-01-24T13:29:40.351 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 97496 ; 2024-01-24T13:29:40.351 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.error_if_exists: 0 2024-01-24T13:29:40.351 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.create_if_missing: 0 2024-01-24T13:29:40.351 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.paranoid_checks: 1 2024-01-24T13:29:40.351 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.flush_verify_memtable_count: 1 2024-01-24T13:29:40.351 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-01-24T13:29:40.351 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-01-24T13:29:40.351 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.env: 0x56167acbc720 2024-01-24T13:29:40.351 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.fs: PosixFileSystem 2024-01-24T13:29:40.351 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.info_log: 0x56167cbb7460 2024-01-24T13:29:40.351 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.max_file_opening_threads: 16 2024-01-24T13:29:40.351 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.statistics: (nil) 2024-01-24T13:29:40.352 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.use_fsync: 0 2024-01-24T13:29:40.352 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.max_log_file_size: 0 2024-01-24T13:29:40.352 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-24T13:29:40.352 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.log_file_time_to_roll: 0 2024-01-24T13:29:40.352 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.keep_log_file_num: 1000 2024-01-24T13:29:40.352 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.recycle_log_file_num: 0 2024-01-24T13:29:40.352 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.allow_fallocate: 1 2024-01-24T13:29:40.352 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.allow_mmap_reads: 0 2024-01-24T13:29:40.352 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.allow_mmap_writes: 0 2024-01-24T13:29:40.352 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.use_direct_reads: 0 2024-01-24T13:29:40.352 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-24T13:29:40.352 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.create_missing_column_families: 0 2024-01-24T13:29:40.352 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.db_log_dir: 2024-01-24T13:29:40.352 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.wal_dir: 2024-01-24T13:29:40.353 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.table_cache_numshardbits: 6 2024-01-24T13:29:40.353 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.WAL_ttl_seconds: 0 2024-01-24T13:29:40.353 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.WAL_size_limit_MB: 0 2024-01-24T13:29:40.353 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-24T13:29:40.353 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-24T13:29:40.353 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.is_fd_close_on_exec: 1 2024-01-24T13:29:40.353 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.advise_random_on_open: 1 2024-01-24T13:29:40.353 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.db_write_buffer_size: 0 2024-01-24T13:29:40.353 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.write_buffer_manager: 0x56167be48320 2024-01-24T13:29:40.353 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-24T13:29:40.353 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-24T13:29:40.353 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.use_adaptive_mutex: 0 2024-01-24T13:29:40.353 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.rate_limiter: (nil) 2024-01-24T13:29:40.353 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-24T13:29:40.354 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.wal_recovery_mode: 2 2024-01-24T13:29:40.354 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.enable_thread_tracking: 0 2024-01-24T13:29:40.354 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.enable_pipelined_write: 0 2024-01-24T13:29:40.354 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.unordered_write: 0 2024-01-24T13:29:40.354 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-24T13:29:40.354 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-24T13:29:40.354 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-24T13:29:40.354 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-24T13:29:40.354 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.row_cache: None 2024-01-24T13:29:40.354 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.wal_filter: None 2024-01-24T13:29:40.354 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-24T13:29:40.355 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.allow_ingest_behind: 0 2024-01-24T13:29:40.355 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.two_write_queues: 0 2024-01-24T13:29:40.355 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.manual_wal_flush: 0 2024-01-24T13:29:40.355 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.wal_compression: 0 2024-01-24T13:29:40.355 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.atomic_flush: 0 2024-01-24T13:29:40.355 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-24T13:29:40.355 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.persist_stats_to_disk: 0 2024-01-24T13:29:40.355 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.write_dbid_to_manifest: 0 2024-01-24T13:29:40.355 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.log_readahead_size: 0 2024-01-24T13:29:40.355 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-01-24T13:29:40.355 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.best_efforts_recovery: 0 2024-01-24T13:29:40.355 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-01-24T13:29:40.355 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-01-24T13:29:40.355 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.allow_data_in_errors: 0 2024-01-24T13:29:40.355 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.db_host_id: __hostname__ 2024-01-24T13:29:40.356 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.enforce_single_del_contracts: true 2024-01-24T13:29:40.356 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.max_background_jobs: 2 2024-01-24T13:29:40.356 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.max_background_compactions: -1 2024-01-24T13:29:40.356 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.max_subcompactions: 1 2024-01-24T13:29:40.356 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-24T13:29:40.356 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-24T13:29:40.356 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.delayed_write_rate : 16777216 2024-01-24T13:29:40.356 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.max_total_wal_size: 0 2024-01-24T13:29:40.356 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-24T13:29:40.356 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.stats_dump_period_sec: 600 2024-01-24T13:29:40.356 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.stats_persist_period_sec: 600 2024-01-24T13:29:40.356 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-24T13:29:40.356 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.max_open_files: -1 2024-01-24T13:29:40.356 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.bytes_per_sync: 0 2024-01-24T13:29:40.357 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.wal_bytes_per_sync: 0 2024-01-24T13:29:40.357 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.strict_bytes_per_sync: 0 2024-01-24T13:29:40.357 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.compaction_readahead_size: 0 2024-01-24T13:29:40.357 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.max_background_flushes: -1 2024-01-24T13:29:40.357 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Compression algorithms supported: 2024-01-24T13:29:40.357 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-24T13:29:40.357 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: kZSTD supported: 0 2024-01-24T13:29:40.357 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: kXpressCompression supported: 0 2024-01-24T13:29:40.357 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: kLZ4HCCompression supported: 1 2024-01-24T13:29:40.357 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: kZlibCompression supported: 1 2024-01-24T13:29:40.357 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: kSnappyCompression supported: 1 2024-01-24T13:29:40.357 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: kLZ4Compression supported: 1 2024-01-24T13:29:40.357 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: kBZip2Compression supported: 0 2024-01-24T13:29:40.357 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Fast CRC32 supported: Supported on x86 2024-01-24T13:29:40.357 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: DMutex implementation: pthread_mutex_t 2024-01-24T13:29:40.358 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-01-24T13:29:40.358 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-01-24T13:29:40.358 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-24T13:29:40.358 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.merge_operator: 2024-01-24T13:29:40.358 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.compaction_filter: None 2024-01-24T13:29:40.358 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.compaction_filter_factory: None 2024-01-24T13:29:40.358 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.sst_partitioner_factory: None 2024-01-24T13:29:40.358 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.memtable_factory: SkipListFactory 2024-01-24T13:29:40.358 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.table_factory: BlockBasedTable 2024-01-24T13:29:40.358 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56167cbb7560) 2024-01-24T13:29:40.358 INFO:journalctl@ceph.mon.a.smithi029.stdout: cache_index_and_filter_blocks: 1 2024-01-24T13:29:40.358 INFO:journalctl@ceph.mon.a.smithi029.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-24T13:29:40.358 INFO:journalctl@ceph.mon.a.smithi029.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-24T13:29:40.358 INFO:journalctl@ceph.mon.a.smithi029.stdout: pin_top_level_index_and_filter: 1 2024-01-24T13:29:40.359 INFO:journalctl@ceph.mon.a.smithi029.stdout: index_type: 0 2024-01-24T13:29:40.359 INFO:journalctl@ceph.mon.a.smithi029.stdout: data_block_index_type: 0 2024-01-24T13:29:40.359 INFO:journalctl@ceph.mon.a.smithi029.stdout: index_shortening: 1 2024-01-24T13:29:40.359 INFO:journalctl@ceph.mon.a.smithi029.stdout: data_block_hash_table_util_ratio: 0.750000 2024-01-24T13:29:40.359 INFO:journalctl@ceph.mon.a.smithi029.stdout: checksum: 4 2024-01-24T13:29:40.359 INFO:journalctl@ceph.mon.a.smithi029.stdout: no_block_cache: 0 2024-01-24T13:29:40.359 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_cache: 0x56167bec91f0 2024-01-24T13:29:40.359 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_cache_name: BinnedLRUCache 2024-01-24T13:29:40.359 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_cache_options: 2024-01-24T13:29:40.359 INFO:journalctl@ceph.mon.a.smithi029.stdout: capacity : 536870912 2024-01-24T13:29:40.359 INFO:journalctl@ceph.mon.a.smithi029.stdout: num_shard_bits : 4 2024-01-24T13:29:40.359 INFO:journalctl@ceph.mon.a.smithi029.stdout: strict_capacity_limit : 0 2024-01-24T13:29:40.359 INFO:journalctl@ceph.mon.a.smithi029.stdout: high_pri_pool_ratio: 0.000 2024-01-24T13:29:40.359 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_cache_compressed: (nil) 2024-01-24T13:29:40.359 INFO:journalctl@ceph.mon.a.smithi029.stdout: persistent_cache: (nil) 2024-01-24T13:29:40.360 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_size: 4096 2024-01-24T13:29:40.360 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_size_deviation: 10 2024-01-24T13:29:40.360 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_restart_interval: 16 2024-01-24T13:29:40.360 INFO:journalctl@ceph.mon.a.smithi029.stdout: index_block_restart_interval: 1 2024-01-24T13:29:40.360 INFO:journalctl@ceph.mon.a.smithi029.stdout: metadata_block_size: 4096 2024-01-24T13:29:40.360 INFO:journalctl@ceph.mon.a.smithi029.stdout: partition_filters: 0 2024-01-24T13:29:40.360 INFO:journalctl@ceph.mon.a.smithi029.stdout: use_delta_encoding: 1 2024-01-24T13:29:40.360 INFO:journalctl@ceph.mon.a.smithi029.stdout: filter_policy: bloomfilter 2024-01-24T13:29:40.360 INFO:journalctl@ceph.mon.a.smithi029.stdout: whole_key_filtering: 1 2024-01-24T13:29:40.360 INFO:journalctl@ceph.mon.a.smithi029.stdout: verify_compression: 0 2024-01-24T13:29:40.360 INFO:journalctl@ceph.mon.a.smithi029.stdout: read_amp_bytes_per_bit: 0 2024-01-24T13:29:40.360 INFO:journalctl@ceph.mon.a.smithi029.stdout: format_version: 5 2024-01-24T13:29:40.360 INFO:journalctl@ceph.mon.a.smithi029.stdout: enable_index_compression: 1 2024-01-24T13:29:40.360 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_align: 0 2024-01-24T13:29:40.361 INFO:journalctl@ceph.mon.a.smithi029.stdout: max_auto_readahead_size: 262144 2024-01-24T13:29:40.361 INFO:journalctl@ceph.mon.a.smithi029.stdout: prepopulate_block_cache: 0 2024-01-24T13:29:40.361 INFO:journalctl@ceph.mon.a.smithi029.stdout: initial_auto_readahead_size: 8192 2024-01-24T13:29:40.361 INFO:journalctl@ceph.mon.a.smithi029.stdout: num_file_reads_for_auto_readahead: 2 2024-01-24T13:29:40.361 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.write_buffer_size: 33554432 2024-01-24T13:29:40.361 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.max_write_buffer_number: 2 2024-01-24T13:29:40.361 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.compression: NoCompression 2024-01-24T13:29:40.361 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.bottommost_compression: Disabled 2024-01-24T13:29:40.361 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.prefix_extractor: nullptr 2024-01-24T13:29:40.361 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-24T13:29:40.361 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.num_levels: 7 2024-01-24T13:29:40.361 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-24T13:29:40.361 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-24T13:29:40.361 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-24T13:29:40.361 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-24T13:29:40.362 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-24T13:29:40.362 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-24T13:29:40.362 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-24T13:29:40.362 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-24T13:29:40.362 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-01-24T13:29:40.362 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-24T13:29:40.362 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-01-24T13:29:40.362 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-01-24T13:29:40.362 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.compression_opts.window_bits: -14 2024-01-24T13:29:40.362 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.compression_opts.level: 32767 2024-01-24T13:29:40.362 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.compression_opts.strategy: 0 2024-01-24T13:29:40.362 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-24T13:29:40.362 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-24T13:29:40.362 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-01-24T13:29:40.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-01-24T13:29:40.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.compression_opts.enabled: false 2024-01-24T13:29:40.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-01-24T13:29:40.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-24T13:29:40.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-24T13:29:40.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-24T13:29:40.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.target_file_size_base: 67108864 2024-01-24T13:29:40.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.target_file_size_multiplier: 1 2024-01-24T13:29:40.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-24T13:29:40.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-24T13:29:40.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-24T13:29:40.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-24T13:29:40.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-24T13:29:40.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-24T13:29:40.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-24T13:29:40.364 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-24T13:29:40.364 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-24T13:29:40.364 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-24T13:29:40.364 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-24T13:29:40.364 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-24T13:29:40.364 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-01-24T13:29:40.364 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.arena_block_size: 1048576 2024-01-24T13:29:40.364 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-24T13:29:40.364 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-24T13:29:40.364 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.disable_auto_compactions: 0 2024-01-24T13:29:40.364 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-24T13:29:40.364 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-24T13:29:40.364 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-24T13:29:40.365 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-24T13:29:40.365 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-24T13:29:40.365 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-24T13:29:40.366 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-24T13:29:40.366 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-24T13:29:40.366 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-24T13:29:40.366 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-24T13:29:40.366 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.table_properties_collectors: 2024-01-24T13:29:40.366 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.inplace_update_support: 0 2024-01-24T13:29:40.366 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.inplace_update_num_locks: 10000 2024-01-24T13:29:40.366 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-24T13:29:40.366 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-24T13:29:40.366 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.memtable_huge_page_size: 0 2024-01-24T13:29:40.366 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.bloom_locality: 0 2024-01-24T13:29:40.366 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.max_successive_merges: 0 2024-01-24T13:29:40.366 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.optimize_filters_for_hits: 0 2024-01-24T13:29:40.366 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.paranoid_file_checks: 0 2024-01-24T13:29:40.366 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.force_consistency_checks: 1 2024-01-24T13:29:40.367 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.report_bg_io_stats: 0 2024-01-24T13:29:40.367 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.ttl: 2592000 2024-01-24T13:29:40.367 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.periodic_compaction_seconds: 0 2024-01-24T13:29:40.367 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-01-24T13:29:40.367 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-01-24T13:29:40.367 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.enable_blob_files: false 2024-01-24T13:29:40.367 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.min_blob_size: 0 2024-01-24T13:29:40.367 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.blob_file_size: 268435456 2024-01-24T13:29:40.367 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.blob_compression_type: NoCompression 2024-01-24T13:29:40.367 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.enable_blob_garbage_collection: false 2024-01-24T13:29:40.367 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-01-24T13:29:40.367 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-01-24T13:29:40.367 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-01-24T13:29:40.367 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.blob_file_starting_level: 0 2024-01-24T13:29:40.367 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-01-24T13:29:40.367 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-01-24T13:29:40.368 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-01-24T13:29:40.368 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 7fc35763-3681-4e1f-9e5a-744b073314da 2024-01-24T13:29:40.369 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: EVENT_LOG_v1 {"time_micros": 1706102979985932, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-01-24T13:29:40.369 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-01-24T13:29:40.369 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: EVENT_LOG_v1 {"time_micros": 1706102979987063, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 92747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 314, "table_properties": {"data_size": 90742, "index_size": 287, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13488, "raw_average_key_size": 50, "raw_value_size": 83355, "raw_average_value_size": 309, "num_data_blocks": 11, "num_entries": 269, "num_filter_entries": 269, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1706102979, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "7fc35763-3681-4e1f-9e5a-744b073314da", "db_session_id": "BWI8OTCDEHQ9RF7JWUG8", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-01-24T13:29:40.369 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: EVENT_LOG_v1 {"time_micros": 1706102979987151, "job": 1, "event": "recovery_finished"} 2024-01-24T13:29:40.369 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-01-24T13:29:40.369 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-01-24T13:29:40.369 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56167bf66000 2024-01-24T13:29:40.369 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: DB pointer 0x56167bf54000 2024-01-24T13:29:40.369 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-01-24T13:29:40.369 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-01-24T13:29:40.369 INFO:journalctl@ceph.mon.a.smithi029.stdout: ** DB Stats ** 2024-01-24T13:29:40.369 INFO:journalctl@ceph.mon.a.smithi029.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-01-24T13:29:40.369 INFO:journalctl@ceph.mon.a.smithi029.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-24T13:29:40.370 INFO:journalctl@ceph.mon.a.smithi029.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-24T13:29:40.370 INFO:journalctl@ceph.mon.a.smithi029.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-24T13:29:40.370 INFO:journalctl@ceph.mon.a.smithi029.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-24T13:29:40.370 INFO:journalctl@ceph.mon.a.smithi029.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-24T13:29:40.370 INFO:journalctl@ceph.mon.a.smithi029.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-24T13:29:40.370 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-01-24T13:29:40.370 INFO:journalctl@ceph.mon.a.smithi029.stdout: ** Compaction Stats [default] ** 2024-01-24T13:29:40.370 INFO:journalctl@ceph.mon.a.smithi029.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-01-24T13:29:40.370 INFO:journalctl@ceph.mon.a.smithi029.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-01-24T13:29:40.370 INFO:journalctl@ceph.mon.a.smithi029.stdout: L0 2/0 92.41 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 127.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-01-24T13:29:40.370 INFO:journalctl@ceph.mon.a.smithi029.stdout: Sum 2/0 92.41 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 127.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-01-24T13:29:40.370 INFO:journalctl@ceph.mon.a.smithi029.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 127.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-01-24T13:29:40.370 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-01-24T13:29:40.371 INFO:journalctl@ceph.mon.a.smithi029.stdout: ** Compaction Stats [default] ** 2024-01-24T13:29:40.371 INFO:journalctl@ceph.mon.a.smithi029.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-01-24T13:29:40.371 INFO:journalctl@ceph.mon.a.smithi029.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-24T13:29:40.371 INFO:journalctl@ceph.mon.a.smithi029.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 127.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-01-24T13:29:40.371 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-01-24T13:29:40.372 INFO:journalctl@ceph.mon.a.smithi029.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-01-24T13:29:40.372 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-01-24T13:29:40.372 INFO:journalctl@ceph.mon.a.smithi029.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-01-24T13:29:40.372 INFO:journalctl@ceph.mon.a.smithi029.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-01-24T13:29:40.372 INFO:journalctl@ceph.mon.a.smithi029.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-24T13:29:40.372 INFO:journalctl@ceph.mon.a.smithi029.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-01-24T13:29:40.372 INFO:journalctl@ceph.mon.a.smithi029.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-01-24T13:29:40.372 INFO:journalctl@ceph.mon.a.smithi029.stdout: AddFile(Keys): cumulative 0, interval 0 2024-01-24T13:29:40.372 INFO:journalctl@ceph.mon.a.smithi029.stdout: Cumulative compaction: 0.00 GB write, 22.09 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-24T13:29:40.372 INFO:journalctl@ceph.mon.a.smithi029.stdout: Interval compaction: 0.00 GB write, 22.09 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-24T13:29:40.372 INFO:journalctl@ceph.mon.a.smithi029.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-24T13:29:40.372 INFO:journalctl@ceph.mon.a.smithi029.stdout: Block cache BinnedLRUCache@0x56167bec91f0#7 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.4e-05 secs_since: 0 2024-01-24T13:29:40.372 INFO:journalctl@ceph.mon.a.smithi029.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-01-24T13:29:40.372 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-01-24T13:29:40.373 INFO:journalctl@ceph.mon.a.smithi029.stdout: ** File Read Latency Histogram By Level [default] ** 2024-01-24T13:29:40.373 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: starting mon.a rank 0 at public addrs [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] at bind addrs [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 83afcf5e-babc-11ee-95b1-87774f69a715 2024-01-24T13:29:40.373 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: mon.a@-1(???) e1 preinit fsid 83afcf5e-babc-11ee-95b1-87774f69a715 2024-01-24T13:29:40.373 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: mon.a@-1(???).mds e1 new map 2024-01-24T13:29:40.373 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: mon.a@-1(???).mds e1 print_map 2024-01-24T13:29:40.373 INFO:journalctl@ceph.mon.a.smithi029.stdout: e1 2024-01-24T13:29:40.373 INFO:journalctl@ceph.mon.a.smithi029.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-01-24T13:29:40.373 INFO:journalctl@ceph.mon.a.smithi029.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-24T13:29:40.373 INFO:journalctl@ceph.mon.a.smithi029.stdout: legacy client fscid: -1 2024-01-24T13:29:40.373 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-01-24T13:29:40.373 INFO:journalctl@ceph.mon.a.smithi029.stdout: No filesystems configured 2024-01-24T13:29:40.373 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-01-24T13:29:40.373 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-24T13:29:40.373 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-24T13:29:40.373 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-24T13:29:40.374 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:39 smithi029 ceph-mon[40651]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-01-24T13:29:40.374 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:40 smithi029 ceph-mon[40651]: mon.a is new leader, mons a in quorum (ranks 0) 2024-01-24T13:29:40.374 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:40 smithi029 ceph-mon[40651]: monmap e1: 1 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-01-24T13:29:40.374 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:40 smithi029 ceph-mon[40651]: fsmap 2024-01-24T13:29:40.374 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:40 smithi029 ceph-mon[40651]: osdmap e1: 0 total, 0 up, 0 in 2024-01-24T13:29:40.374 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:40 smithi029 ceph-mon[40651]: mgrmap e1: no daemons active 2024-01-24T13:29:40.849 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.354+0000 7f9cf0e3b700 1 Processor -- start 2024-01-24T13:29:40.849 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.354+0000 7f9cf0e3b700 1 -- start start 2024-01-24T13:29:40.849 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.354+0000 7f9cf0e3b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cec104780 0x7f9cec104ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:29:40.849 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.354+0000 7f9cf0e3b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cec105170 con 0x7f9cec104780 2024-01-24T13:29:40.849 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.355+0000 7f9cea59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cec104780 0x7f9cec104ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:29:40.849 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.355+0000 7f9cea59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cec104780 0x7f9cec104ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35808/0 (socket says 172.21.15.29:35808) 2024-01-24T13:29:40.850 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.355+0000 7f9cea59c700 1 -- 172.21.15.29:0/3296257752 learned_addr learned my addr 172.21.15.29:0/3296257752 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:29:40.850 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.355+0000 7f9cea59c700 1 -- 172.21.15.29:0/3296257752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cec107710 con 0x7f9cec104780 2024-01-24T13:29:40.850 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.355+0000 7f9cea59c700 1 --2- 172.21.15.29:0/3296257752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cec104780 0x7f9cec104ba0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc0098a0 tx=0x7f9cdc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=547b440d7f75f1a8 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:29:40.850 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.356+0000 7f9ce959a700 1 -- 172.21.15.29:0/3296257752 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9cdc004160 con 0x7f9cec104780 2024-01-24T13:29:40.850 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.356+0000 7f9ce959a700 1 -- 172.21.15.29:0/3296257752 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9cdc0042c0 con 0x7f9cec104780 2024-01-24T13:29:40.850 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.356+0000 7f9ce959a700 1 -- 172.21.15.29:0/3296257752 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9cdc011a00 con 0x7f9cec104780 2024-01-24T13:29:40.850 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.356+0000 7f9cf0e3b700 1 -- 172.21.15.29:0/3296257752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cec104780 msgr2=0x7f9cec104ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:29:40.850 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.356+0000 7f9cf0e3b700 1 --2- 172.21.15.29:0/3296257752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cec104780 0x7f9cec104ba0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc0098a0 tx=0x7f9cdc009bb0 comp rx=0 tx=0).stop 2024-01-24T13:29:40.850 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.356+0000 7f9cf0e3b700 1 -- 172.21.15.29:0/3296257752 shutdown_connections 2024-01-24T13:29:40.850 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.356+0000 7f9cf0e3b700 1 --2- 172.21.15.29:0/3296257752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cec104780 0x7f9cec104ba0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:29:40.850 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.356+0000 7f9cf0e3b700 1 -- 172.21.15.29:0/3296257752 >> 172.21.15.29:0/3296257752 conn(0x7f9cec1000d0 msgr2=0x7f9cec1024f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:29:40.850 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.356+0000 7f9cf0e3b700 1 -- 172.21.15.29:0/3296257752 shutdown_connections 2024-01-24T13:29:40.850 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.356+0000 7f9cf0e3b700 1 -- 172.21.15.29:0/3296257752 wait complete. 2024-01-24T13:29:40.850 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.358+0000 7f9cf0e3b700 1 Processor -- start 2024-01-24T13:29:40.851 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.358+0000 7f9cf0e3b700 1 -- start start 2024-01-24T13:29:40.851 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.358+0000 7f9cf0e3b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cec104780 0x7f9cec107070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:29:40.851 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.358+0000 7f9cf0e3b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cec1075b0 con 0x7f9cec104780 2024-01-24T13:29:40.851 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.358+0000 7f9cea59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cec104780 0x7f9cec107070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:29:40.851 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.358+0000 7f9cea59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cec104780 0x7f9cec107070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35814/0 (socket says 172.21.15.29:35814) 2024-01-24T13:29:40.851 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.358+0000 7f9cea59c700 1 -- 172.21.15.29:0/3587526201 learned_addr learned my addr 172.21.15.29:0/3587526201 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:29:40.851 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.358+0000 7f9cea59c700 1 -- 172.21.15.29:0/3587526201 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cdc009550 con 0x7f9cec104780 2024-01-24T13:29:40.851 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.359+0000 7f9cea59c700 1 --2- 172.21.15.29:0/3587526201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cec104780 0x7f9cec107070 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc009130 tx=0x7f9cdc011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:29:40.851 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.359+0000 7f9cdb7fe700 1 -- 172.21.15.29:0/3587526201 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9cdc003ab0 con 0x7f9cec104780 2024-01-24T13:29:40.851 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.359+0000 7f9cdb7fe700 1 -- 172.21.15.29:0/3587526201 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9cdc003c10 con 0x7f9cec104780 2024-01-24T13:29:40.851 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.359+0000 7f9cf0e3b700 1 -- 172.21.15.29:0/3587526201 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cec1056c0 con 0x7f9cec104780 2024-01-24T13:29:40.851 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.359+0000 7f9cdb7fe700 1 -- 172.21.15.29:0/3587526201 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9cdc0247d0 con 0x7f9cec104780 2024-01-24T13:29:40.851 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.359+0000 7f9cf0e3b700 1 -- 172.21.15.29:0/3587526201 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cec105b60 con 0x7f9cec104780 2024-01-24T13:29:40.851 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.359+0000 7f9cdb7fe700 1 -- 172.21.15.29:0/3587526201 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9cdc024d50 con 0x7f9cec104780 2024-01-24T13:29:40.852 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.359+0000 7f9cdb7fe700 1 -- 172.21.15.29:0/3587526201 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9cdc0193c0 con 0x7f9cec104780 2024-01-24T13:29:40.852 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.360+0000 7f9cf0e3b700 1 -- 172.21.15.29:0/3587526201 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cec062370 con 0x7f9cec104780 2024-01-24T13:29:40.852 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.363+0000 7f9cdb7fe700 1 -- 172.21.15.29:0/3587526201 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f9cdc01abb0 con 0x7f9cec104780 2024-01-24T13:29:40.852 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.435+0000 7f9cf0e3b700 1 -- 172.21.15.29:0/3587526201 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f9cec1061c0 con 0x7f9cec104780 2024-01-24T13:29:40.852 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.437+0000 7f9cdb7fe700 1 -- 172.21.15.29:0/3587526201 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f9cdc01ad50 con 0x7f9cec104780 2024-01-24T13:29:40.852 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.438+0000 7f9cf0e3b700 1 -- 172.21.15.29:0/3587526201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cec104780 msgr2=0x7f9cec107070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:29:40.852 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.438+0000 7f9cf0e3b700 1 --2- 172.21.15.29:0/3587526201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cec104780 0x7f9cec107070 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc009130 tx=0x7f9cdc011ad0 comp rx=0 tx=0).stop 2024-01-24T13:29:40.852 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.438+0000 7f9cf0e3b700 1 -- 172.21.15.29:0/3587526201 shutdown_connections 2024-01-24T13:29:40.852 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.438+0000 7f9cf0e3b700 1 --2- 172.21.15.29:0/3587526201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cec104780 0x7f9cec107070 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:29:40.852 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.438+0000 7f9cf0e3b700 1 -- 172.21.15.29:0/3587526201 >> 172.21.15.29:0/3587526201 conn(0x7f9cec1000d0 msgr2=0x7f9cec101730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:29:40.852 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.438+0000 7f9cf0e3b700 1 -- 172.21.15.29:0/3587526201 shutdown_connections 2024-01-24T13:29:40.852 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:40.438+0000 7f9cf0e3b700 1 -- 172.21.15.29:0/3587526201 wait complete. 2024-01-24T13:29:40.852 INFO:teuthology.orchestra.run.smithi029.stdout:Wrote config to /etc/ceph/ceph.conf 2024-01-24T13:29:40.874 INFO:teuthology.orchestra.run.smithi029.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-01-24T13:29:40.874 INFO:teuthology.orchestra.run.smithi029.stdout:Creating mgr... 2024-01-24T13:29:40.874 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying port 0.0.0.0:9283 ... 2024-01-24T13:29:40.875 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying port 0.0.0.0:8765 ... 2024-01-24T13:29:41.099 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mgr.x 2024-01-24T13:29:41.099 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Failed to reset failed state of unit ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mgr.x.service: Unit ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mgr.x.service not loaded. 2024-01-24T13:29:41.203 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-83afcf5e-babc-11ee-95b1-87774f69a715.target.wants/ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-83afcf5e-babc-11ee-95b1-87774f69a715@.service. 2024-01-24T13:29:41.437 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:29:41 smithi029 systemd[1]: Starting Ceph mgr.x for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:29:41.636 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-01-24T13:29:41.636 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout disabled 2024-01-24T13:29:41.646 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-01-24T13:29:41.646 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-01-24T13:29:41.647 INFO:teuthology.orchestra.run.smithi029.stdout:firewalld.service is not enabled 2024-01-24T13:29:41.647 INFO:teuthology.orchestra.run.smithi029.stdout:Not possible to enable service . firewalld.service is not available 2024-01-24T13:29:41.658 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-01-24T13:29:41.658 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout disabled 2024-01-24T13:29:41.668 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-01-24T13:29:41.668 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-01-24T13:29:41.669 INFO:teuthology.orchestra.run.smithi029.stdout:firewalld.service is not enabled 2024-01-24T13:29:41.669 INFO:teuthology.orchestra.run.smithi029.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-01-24T13:29:41.669 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mgr to start... 2024-01-24T13:29:41.669 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mgr... 2024-01-24T13:29:41.780 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:41 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/3587526201' entity='client.admin' 2024-01-24T13:29:41.780 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:29:41 smithi029 bash[40887]: fb905e9e9fa0b2576df39e811e48cc6cc3a7a3b01ec6c4ffc2c6343f0c4b9cba 2024-01-24T13:29:41.780 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:29:41 smithi029 systemd[1]: Started Ceph mgr.x for 83afcf5e-babc-11ee-95b1-87774f69a715. 2024-01-24T13:29:42.349 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:29:42 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:29:42.089+0000 7fd4c5e38200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-24T13:29:42.349 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:29:42 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:29:42.196+0000 7fd4c5e38200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-24T13:29:42.349 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:29:42 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:29:42.300+0000 7fd4c5e38200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-24T13:29:42.506 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-01-24T13:29:42.506 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-01-24T13:29:42.506 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsid": "83afcf5e-babc-11ee-95b1-87774f69a715", 2024-01-24T13:29:42.506 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "health": { 2024-01-24T13:29:42.506 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-01-24T13:29:42.506 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-01-24T13:29:42.506 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-01-24T13:29:42.506 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-01-24T13:29:42.506 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-01-24T13:29:42.506 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-01-24T13:29:42.507 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 0 2024-01-24T13:29:42.507 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-01-24T13:29:42.507 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-01-24T13:29:42.507 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "a" 2024-01-24T13:29:42.507 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-01-24T13:29:42.507 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-01-24T13:29:42.507 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "monmap": { 2024-01-24T13:29:42.507 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-24T13:29:42.507 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-01-24T13:29:42.507 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-01-24T13:29:42.507 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-01-24T13:29:42.507 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-01-24T13:29:42.507 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-24T13:29:42.507 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-01-24T13:29:42.507 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-01-24T13:29:42.517 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-01-24T13:29:42.517 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-01-24T13:29:42.517 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-01-24T13:29:42.517 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-01-24T13:29:42.517 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-01-24T13:29:42.517 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-01-24T13:29:42.517 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-01-24T13:29:42.517 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-01-24T13:29:42.517 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-01-24T13:29:42.517 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-01-24T13:29:42.517 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-01-24T13:29:42.517 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-01-24T13:29:42.517 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-01-24T13:29:42.518 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-01-24T13:29:42.518 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-01-24T13:29:42.518 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-01-24T13:29:42.518 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-24T13:29:42.518 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-01-24T13:29:42.518 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-01-24T13:29:42.518 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-01-24T13:29:42.518 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-01-24T13:29:42.518 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": false, 2024-01-24T13:29:42.518 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-01-24T13:29:42.518 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modules": [ 2024-01-24T13:29:42.518 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "iostat", 2024-01-24T13:29:42.519 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "nfs", 2024-01-24T13:29:42.519 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "restful" 2024-01-24T13:29:42.519 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-01-24T13:29:42.519 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-01-24T13:29:42.519 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-01-24T13:29:42.519 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-01-24T13:29:42.519 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-24T13:29:42.519 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modified": "2024-01-24T13:29:33.176561+0000", 2024-01-24T13:29:42.519 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-01-24T13:29:42.519 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-01-24T13:29:42.519 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-01-24T13:29:42.519 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-01-24T13:29:42.519 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.064+0000 7f2eaa25b700 1 Processor -- start 2024-01-24T13:29:42.519 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.064+0000 7f2eaa25b700 1 -- start start 2024-01-24T13:29:42.519 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.064+0000 7f2eaa25b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ea4106470 0x7f2ea4106890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:29:42.520 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.064+0000 7f2eaa25b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea4106e60 con 0x7f2ea4106470 2024-01-24T13:29:42.520 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.065+0000 7f2ea37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ea4106470 0x7f2ea4106890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:29:42.520 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.065+0000 7f2ea37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ea4106470 0x7f2ea4106890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58034/0 (socket says 172.21.15.29:58034) 2024-01-24T13:29:42.520 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.065+0000 7f2ea37fe700 1 -- 172.21.15.29:0/3211413651 learned_addr learned my addr 172.21.15.29:0/3211413651 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:29:42.520 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.065+0000 7f2ea37fe700 1 -- 172.21.15.29:0/3211413651 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea4107670 con 0x7f2ea4106470 2024-01-24T13:29:42.520 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.065+0000 7f2ea37fe700 1 --2- 172.21.15.29:0/3211413651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ea4106470 0x7f2ea4106890 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2e940098a0 tx=0x7f2e94009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dda76032b8aaa42 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:29:42.520 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.066+0000 7f2ea27fc700 1 -- 172.21.15.29:0/3211413651 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e94004160 con 0x7f2ea4106470 2024-01-24T13:29:42.520 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.066+0000 7f2ea27fc700 1 -- 172.21.15.29:0/3211413651 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2e940042c0 con 0x7f2ea4106470 2024-01-24T13:29:42.520 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.066+0000 7f2ea27fc700 1 -- 172.21.15.29:0/3211413651 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e94011a00 con 0x7f2ea4106470 2024-01-24T13:29:42.520 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.066+0000 7f2eaa25b700 1 -- 172.21.15.29:0/3211413651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ea4106470 msgr2=0x7f2ea4106890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:29:42.520 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.066+0000 7f2eaa25b700 1 --2- 172.21.15.29:0/3211413651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ea4106470 0x7f2ea4106890 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2e940098a0 tx=0x7f2e94009bb0 comp rx=0 tx=0).stop 2024-01-24T13:29:42.520 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.066+0000 7f2eaa25b700 1 -- 172.21.15.29:0/3211413651 shutdown_connections 2024-01-24T13:29:42.520 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.066+0000 7f2eaa25b700 1 --2- 172.21.15.29:0/3211413651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ea4106470 0x7f2ea4106890 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:29:42.521 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.066+0000 7f2eaa25b700 1 -- 172.21.15.29:0/3211413651 >> 172.21.15.29:0/3211413651 conn(0x7f2ea4101a50 msgr2=0x7f2ea4103e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:29:42.521 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.067+0000 7f2eaa25b700 1 -- 172.21.15.29:0/3211413651 shutdown_connections 2024-01-24T13:29:42.521 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.067+0000 7f2eaa25b700 1 -- 172.21.15.29:0/3211413651 wait complete. 2024-01-24T13:29:42.521 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.067+0000 7f2eaa25b700 1 Processor -- start 2024-01-24T13:29:42.521 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.067+0000 7f2eaa25b700 1 -- start start 2024-01-24T13:29:42.521 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.068+0000 7f2eaa25b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ea4106470 0x7f2ea407bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:29:42.521 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.068+0000 7f2eaa25b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea407c170 con 0x7f2ea4106470 2024-01-24T13:29:42.521 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.068+0000 7f2ea37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ea4106470 0x7f2ea407bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:29:42.521 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.068+0000 7f2ea37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ea4106470 0x7f2ea407bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58036/0 (socket says 172.21.15.29:58036) 2024-01-24T13:29:42.521 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.068+0000 7f2ea37fe700 1 -- 172.21.15.29:0/330120735 learned_addr learned my addr 172.21.15.29:0/330120735 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:29:42.521 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.068+0000 7f2ea37fe700 1 -- 172.21.15.29:0/330120735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e94009550 con 0x7f2ea4106470 2024-01-24T13:29:42.521 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.068+0000 7f2ea37fe700 1 --2- 172.21.15.29:0/330120735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ea4106470 0x7f2ea407bc30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2e94009fb0 tx=0x7f2e94011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:29:42.521 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.068+0000 7f2ea0ff9700 1 -- 172.21.15.29:0/330120735 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e94003930 con 0x7f2ea4106470 2024-01-24T13:29:42.521 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.068+0000 7f2ea0ff9700 1 -- 172.21.15.29:0/330120735 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2e94003a90 con 0x7f2ea4106470 2024-01-24T13:29:42.522 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.068+0000 7f2eaa25b700 1 -- 172.21.15.29:0/330120735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ea407a280 con 0x7f2ea4106470 2024-01-24T13:29:42.522 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.069+0000 7f2ea0ff9700 1 -- 172.21.15.29:0/330120735 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e940195e0 con 0x7f2ea4106470 2024-01-24T13:29:42.522 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.069+0000 7f2eaa25b700 1 -- 172.21.15.29:0/330120735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ea407a720 con 0x7f2ea4106470 2024-01-24T13:29:42.522 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.069+0000 7f2ea0ff9700 1 -- 172.21.15.29:0/330120735 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f2e94019870 con 0x7f2ea4106470 2024-01-24T13:29:42.522 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.069+0000 7f2ea0ff9700 1 -- 172.21.15.29:0/330120735 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2e9402da10 con 0x7f2ea4106470 2024-01-24T13:29:42.522 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.070+0000 7f2eaa25b700 1 -- 172.21.15.29:0/330120735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ea4062370 con 0x7f2ea4106470 2024-01-24T13:29:42.522 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.072+0000 7f2ea0ff9700 1 -- 172.21.15.29:0/330120735 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f2e94003c00 con 0x7f2ea4106470 2024-01-24T13:29:42.522 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.148+0000 7f2eaa25b700 1 -- 172.21.15.29:0/330120735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f2ea410a860 con 0x7f2ea4106470 2024-01-24T13:29:42.522 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.149+0000 7f2ea0ff9700 1 -- 172.21.15.29:0/330120735 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f2e9402d440 con 0x7f2ea4106470 2024-01-24T13:29:42.522 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.150+0000 7f2eaa25b700 1 -- 172.21.15.29:0/330120735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ea4106470 msgr2=0x7f2ea407bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:29:42.523 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.150+0000 7f2eaa25b700 1 --2- 172.21.15.29:0/330120735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ea4106470 0x7f2ea407bc30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2e94009fb0 tx=0x7f2e94011ad0 comp rx=0 tx=0).stop 2024-01-24T13:29:42.523 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.150+0000 7f2eaa25b700 1 -- 172.21.15.29:0/330120735 shutdown_connections 2024-01-24T13:29:42.523 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.150+0000 7f2eaa25b700 1 --2- 172.21.15.29:0/330120735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ea4106470 0x7f2ea407bc30 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:29:42.523 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.150+0000 7f2eaa25b700 1 -- 172.21.15.29:0/330120735 >> 172.21.15.29:0/330120735 conn(0x7f2ea4101a50 msgr2=0x7f2ea41030e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:29:42.523 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.150+0000 7f2eaa25b700 1 -- 172.21.15.29:0/330120735 shutdown_connections 2024-01-24T13:29:42.523 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:42.150+0000 7f2eaa25b700 1 -- 172.21.15.29:0/330120735 wait complete. 2024-01-24T13:29:42.523 INFO:teuthology.orchestra.run.smithi029.stdout:mgr not available, waiting (1/15)... 2024-01-24T13:29:42.850 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:42 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/330120735' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-24T13:29:42.850 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:29:42 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:29:42.396+0000 7fd4c5e38200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-24T13:29:43.379 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:29:43 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:29:43.108+0000 7fd4c5e38200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-24T13:29:43.789 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:29:43 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:29:43.378+0000 7fd4c5e38200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-24T13:29:43.789 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:29:43 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:29:43.473+0000 7fd4c5e38200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-24T13:29:43.789 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:29:43 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:29:43.596+0000 7fd4c5e38200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-24T13:29:43.789 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:29:43 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:29:43.788+0000 7fd4c5e38200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-24T13:29:44.099 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:29:43 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:29:43.931+0000 7fd4c5e38200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-24T13:29:44.099 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:29:44 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:29:44.026+0000 7fd4c5e38200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-24T13:29:44.754 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:29:44 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:29:44.447+0000 7fd4c5e38200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-24T13:29:45.012 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:29:44 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:29:44.753+0000 7fd4c5e38200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-24T13:29:45.272 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:45 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/2395550207' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-24T13:29:45.272 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:29:45 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:29:45.083+0000 7fd4c5e38200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-24T13:29:45.272 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:29:45 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:29:45.190+0000 7fd4c5e38200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-24T13:29:45.355 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-01-24T13:29:45.356 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-01-24T13:29:45.356 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsid": "83afcf5e-babc-11ee-95b1-87774f69a715", 2024-01-24T13:29:45.356 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "health": { 2024-01-24T13:29:45.356 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-01-24T13:29:45.356 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-01-24T13:29:45.356 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-01-24T13:29:45.356 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-01-24T13:29:45.356 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-01-24T13:29:45.356 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-01-24T13:29:45.356 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 0 2024-01-24T13:29:45.356 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-01-24T13:29:45.356 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-01-24T13:29:45.356 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "a" 2024-01-24T13:29:45.357 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-01-24T13:29:45.357 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_age": 4, 2024-01-24T13:29:45.357 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "monmap": { 2024-01-24T13:29:45.357 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-24T13:29:45.357 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-01-24T13:29:45.357 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-01-24T13:29:45.357 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-01-24T13:29:45.357 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-01-24T13:29:45.357 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-24T13:29:45.357 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-01-24T13:29:45.357 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-01-24T13:29:45.357 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-01-24T13:29:45.357 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-01-24T13:29:45.357 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-01-24T13:29:45.357 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-01-24T13:29:45.358 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-01-24T13:29:45.358 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-01-24T13:29:45.358 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-01-24T13:29:45.358 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-01-24T13:29:45.358 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-01-24T13:29:45.358 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-01-24T13:29:45.358 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-01-24T13:29:45.358 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-01-24T13:29:45.358 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-01-24T13:29:45.358 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-01-24T13:29:45.358 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-01-24T13:29:45.358 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-01-24T13:29:45.358 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-24T13:29:45.358 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-01-24T13:29:45.359 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-01-24T13:29:45.359 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-01-24T13:29:45.359 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-01-24T13:29:45.359 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": false, 2024-01-24T13:29:45.359 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-01-24T13:29:45.359 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modules": [ 2024-01-24T13:29:45.359 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "iostat", 2024-01-24T13:29:45.359 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "nfs", 2024-01-24T13:29:45.359 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "restful" 2024-01-24T13:29:45.359 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-01-24T13:29:45.359 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-01-24T13:29:45.359 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-01-24T13:29:45.359 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-01-24T13:29:45.359 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-24T13:29:45.359 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modified": "2024-01-24T13:29:33.176561+0000", 2024-01-24T13:29:45.360 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-01-24T13:29:45.360 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-01-24T13:29:45.360 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-01-24T13:29:45.360 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-01-24T13:29:45.360 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.900+0000 7fddf9e4f700 1 Processor -- start 2024-01-24T13:29:45.360 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.900+0000 7fddf9e4f700 1 -- start start 2024-01-24T13:29:45.360 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.900+0000 7fddf9e4f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddf4105360 0x7fddf4105780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:29:45.360 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.900+0000 7fddf9e4f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf4105d50 con 0x7fddf4105360 2024-01-24T13:29:45.360 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.900+0000 7fddf37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddf4105360 0x7fddf4105780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:29:45.361 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.901+0000 7fddf37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddf4105360 0x7fddf4105780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58040/0 (socket says 172.21.15.29:58040) 2024-01-24T13:29:45.361 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.901+0000 7fddf37fe700 1 -- 172.21.15.29:0/1570451950 learned_addr learned my addr 172.21.15.29:0/1570451950 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:29:45.361 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.901+0000 7fddf37fe700 1 -- 172.21.15.29:0/1570451950 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf4106560 con 0x7fddf4105360 2024-01-24T13:29:45.361 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.901+0000 7fddf37fe700 1 --2- 172.21.15.29:0/1570451950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddf4105360 0x7fddf4105780 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdde40098a0 tx=0x7fdde4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d5dc9e7bb6fdde7 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:29:45.361 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.901+0000 7fddf27fc700 1 -- 172.21.15.29:0/1570451950 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdde4004160 con 0x7fddf4105360 2024-01-24T13:29:45.361 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.901+0000 7fddf27fc700 1 -- 172.21.15.29:0/1570451950 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdde40042c0 con 0x7fddf4105360 2024-01-24T13:29:45.361 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.901+0000 7fddf27fc700 1 -- 172.21.15.29:0/1570451950 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdde4011a00 con 0x7fddf4105360 2024-01-24T13:29:45.361 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.902+0000 7fddf9e4f700 1 -- 172.21.15.29:0/1570451950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddf4105360 msgr2=0x7fddf4105780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:29:45.361 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.902+0000 7fddf9e4f700 1 --2- 172.21.15.29:0/1570451950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddf4105360 0x7fddf4105780 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdde40098a0 tx=0x7fdde4009bb0 comp rx=0 tx=0).stop 2024-01-24T13:29:45.361 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.902+0000 7fddf9e4f700 1 -- 172.21.15.29:0/1570451950 shutdown_connections 2024-01-24T13:29:45.361 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.902+0000 7fddf9e4f700 1 --2- 172.21.15.29:0/1570451950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddf4105360 0x7fddf4105780 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:29:45.361 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.902+0000 7fddf9e4f700 1 -- 172.21.15.29:0/1570451950 >> 172.21.15.29:0/1570451950 conn(0x7fddf4100980 msgr2=0x7fddf4102da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:29:45.361 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.902+0000 7fddf9e4f700 1 -- 172.21.15.29:0/1570451950 shutdown_connections 2024-01-24T13:29:45.362 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.902+0000 7fddf9e4f700 1 -- 172.21.15.29:0/1570451950 wait complete. 2024-01-24T13:29:45.362 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.903+0000 7fddf9e4f700 1 Processor -- start 2024-01-24T13:29:45.362 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.903+0000 7fddf9e4f700 1 -- start start 2024-01-24T13:29:45.362 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.903+0000 7fddf9e4f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddf4105360 0x7fddf41904f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:29:45.362 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.903+0000 7fddf9e4f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf4190a30 con 0x7fddf4105360 2024-01-24T13:29:45.362 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.903+0000 7fddf37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddf4105360 0x7fddf41904f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:29:45.362 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.903+0000 7fddf37fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddf4105360 0x7fddf41904f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58050/0 (socket says 172.21.15.29:58050) 2024-01-24T13:29:45.362 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.903+0000 7fddf37fe700 1 -- 172.21.15.29:0/2395550207 learned_addr learned my addr 172.21.15.29:0/2395550207 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:29:45.362 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.904+0000 7fddf37fe700 1 -- 172.21.15.29:0/2395550207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde4009550 con 0x7fddf4105360 2024-01-24T13:29:45.362 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.904+0000 7fddf37fe700 1 --2- 172.21.15.29:0/2395550207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddf4105360 0x7fddf41904f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fdde4006af0 tx=0x7fdde4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:29:45.362 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.904+0000 7fddf0ff9700 1 -- 172.21.15.29:0/2395550207 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdde4003a40 con 0x7fddf4105360 2024-01-24T13:29:45.362 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.904+0000 7fddf0ff9700 1 -- 172.21.15.29:0/2395550207 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdde4003ba0 con 0x7fddf4105360 2024-01-24T13:29:45.362 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.904+0000 7fddf0ff9700 1 -- 172.21.15.29:0/2395550207 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdde401a700 con 0x7fddf4105360 2024-01-24T13:29:45.362 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.904+0000 7fddf9e4f700 1 -- 172.21.15.29:0/2395550207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddf418eb40 con 0x7fddf4105360 2024-01-24T13:29:45.363 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.904+0000 7fddf9e4f700 1 -- 172.21.15.29:0/2395550207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddf418efe0 con 0x7fddf4105360 2024-01-24T13:29:45.363 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.905+0000 7fddf0ff9700 1 -- 172.21.15.29:0/2395550207 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fdde401ac80 con 0x7fddf4105360 2024-01-24T13:29:45.363 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.905+0000 7fddf0ff9700 1 -- 172.21.15.29:0/2395550207 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fdde4019700 con 0x7fddf4105360 2024-01-24T13:29:45.363 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.905+0000 7fddf9e4f700 1 -- 172.21.15.29:0/2395550207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddf4062370 con 0x7fddf4105360 2024-01-24T13:29:45.363 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.908+0000 7fddf0ff9700 1 -- 172.21.15.29:0/2395550207 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fdde4024ad0 con 0x7fddf4105360 2024-01-24T13:29:45.363 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.986+0000 7fddf9e4f700 1 -- 172.21.15.29:0/2395550207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fddf418f640 con 0x7fddf4105360 2024-01-24T13:29:45.363 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.987+0000 7fddf0ff9700 1 -- 172.21.15.29:0/2395550207 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7fddf418f640 con 0x7fddf4105360 2024-01-24T13:29:45.363 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.988+0000 7fddf9e4f700 1 -- 172.21.15.29:0/2395550207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddf4105360 msgr2=0x7fddf41904f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:29:45.363 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.988+0000 7fddf9e4f700 1 --2- 172.21.15.29:0/2395550207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddf4105360 0x7fddf41904f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fdde4006af0 tx=0x7fdde4011ad0 comp rx=0 tx=0).stop 2024-01-24T13:29:45.363 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.988+0000 7fddf9e4f700 1 -- 172.21.15.29:0/2395550207 shutdown_connections 2024-01-24T13:29:45.363 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.988+0000 7fddf9e4f700 1 --2- 172.21.15.29:0/2395550207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddf4105360 0x7fddf41904f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:29:45.363 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.988+0000 7fddf9e4f700 1 -- 172.21.15.29:0/2395550207 >> 172.21.15.29:0/2395550207 conn(0x7fddf4100980 msgr2=0x7fddf4079860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:29:45.363 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.988+0000 7fddf9e4f700 1 -- 172.21.15.29:0/2395550207 shutdown_connections 2024-01-24T13:29:45.363 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:44.988+0000 7fddf9e4f700 1 -- 172.21.15.29:0/2395550207 wait complete. 2024-01-24T13:29:45.363 INFO:teuthology.orchestra.run.smithi029.stdout:mgr not available, waiting (2/15)... 2024-01-24T13:29:46.316 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:29:46 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:29:46.044+0000 7fd4c5e38200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-24T13:29:46.317 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:29:46 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:29:46.149+0000 7fd4c5e38200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-24T13:29:46.599 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:29:46 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:29:46.315+0000 7fd4c5e38200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-24T13:29:47.349 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:29:47 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:29:47.033+0000 7fd4c5e38200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-24T13:29:47.349 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:29:47 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:29:47.235+0000 7fd4c5e38200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-24T13:29:47.817 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:29:47 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:29:47.457+0000 7fd4c5e38200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-24T13:29:48.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:47 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/3964080341' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-24T13:29:48.155 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-01-24T13:29:48.155 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-01-24T13:29:48.155 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsid": "83afcf5e-babc-11ee-95b1-87774f69a715", 2024-01-24T13:29:48.155 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "health": { 2024-01-24T13:29:48.155 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-01-24T13:29:48.155 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-01-24T13:29:48.155 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-01-24T13:29:48.155 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-01-24T13:29:48.155 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-01-24T13:29:48.155 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-01-24T13:29:48.155 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 0 2024-01-24T13:29:48.155 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-01-24T13:29:48.156 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-01-24T13:29:48.156 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "a" 2024-01-24T13:29:48.156 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-01-24T13:29:48.156 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-01-24T13:29:48.156 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "monmap": { 2024-01-24T13:29:48.156 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-24T13:29:48.156 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-01-24T13:29:48.156 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-01-24T13:29:48.156 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-01-24T13:29:48.156 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-01-24T13:29:48.156 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-24T13:29:48.156 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-01-24T13:29:48.156 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-01-24T13:29:48.167 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-01-24T13:29:48.167 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-01-24T13:29:48.168 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-01-24T13:29:48.168 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-01-24T13:29:48.168 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-01-24T13:29:48.168 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-01-24T13:29:48.168 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-01-24T13:29:48.168 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-01-24T13:29:48.168 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-01-24T13:29:48.168 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-01-24T13:29:48.168 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-01-24T13:29:48.168 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-01-24T13:29:48.168 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-01-24T13:29:48.168 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-01-24T13:29:48.168 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-01-24T13:29:48.168 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-01-24T13:29:48.168 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-24T13:29:48.169 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-01-24T13:29:48.169 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-01-24T13:29:48.169 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-01-24T13:29:48.169 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-01-24T13:29:48.169 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": false, 2024-01-24T13:29:48.169 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-01-24T13:29:48.169 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modules": [ 2024-01-24T13:29:48.169 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "iostat", 2024-01-24T13:29:48.169 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "nfs", 2024-01-24T13:29:48.169 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "restful" 2024-01-24T13:29:48.169 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-01-24T13:29:48.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-01-24T13:29:48.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-01-24T13:29:48.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-01-24T13:29:48.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-24T13:29:48.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modified": "2024-01-24T13:29:33.176561+0000", 2024-01-24T13:29:48.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-01-24T13:29:48.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-01-24T13:29:48.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-01-24T13:29:48.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-01-24T13:29:48.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.693+0000 7fe294928700 1 Processor -- start 2024-01-24T13:29:48.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.693+0000 7fe294928700 1 -- start start 2024-01-24T13:29:48.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.694+0000 7fe294928700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe290104d80 0x7fe2901051a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:29:48.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.694+0000 7fe294928700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe290079e10 con 0x7fe290104d80 2024-01-24T13:29:48.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.694+0000 7fe28e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe290104d80 0x7fe2901051a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:29:48.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.694+0000 7fe28e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe290104d80 0x7fe2901051a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58058/0 (socket says 172.21.15.29:58058) 2024-01-24T13:29:48.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.694+0000 7fe28e59c700 1 -- 172.21.15.29:0/1553959812 learned_addr learned my addr 172.21.15.29:0/1553959812 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:29:48.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.694+0000 7fe28e59c700 1 -- 172.21.15.29:0/1553959812 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2901056e0 con 0x7fe290104d80 2024-01-24T13:29:48.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.694+0000 7fe28e59c700 1 --2- 172.21.15.29:0/1553959812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe290104d80 0x7fe2901051a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe280009e80 tx=0x7fe28000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4d011fd5d5d0480 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:29:48.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.694+0000 7fe28d59a700 1 -- 172.21.15.29:0/1553959812 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe28000e070 con 0x7fe290104d80 2024-01-24T13:29:48.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.695+0000 7fe28d59a700 1 -- 172.21.15.29:0/1553959812 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe28000b8c0 con 0x7fe290104d80 2024-01-24T13:29:48.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.695+0000 7fe28d59a700 1 -- 172.21.15.29:0/1553959812 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe2800044c0 con 0x7fe290104d80 2024-01-24T13:29:48.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.695+0000 7fe294928700 1 -- 172.21.15.29:0/1553959812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe290104d80 msgr2=0x7fe2901051a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:29:48.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.695+0000 7fe294928700 1 --2- 172.21.15.29:0/1553959812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe290104d80 0x7fe2901051a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe280009e80 tx=0x7fe28000b330 comp rx=0 tx=0).stop 2024-01-24T13:29:48.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.695+0000 7fe294928700 1 -- 172.21.15.29:0/1553959812 shutdown_connections 2024-01-24T13:29:48.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.695+0000 7fe294928700 1 --2- 172.21.15.29:0/1553959812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe290104d80 0x7fe2901051a0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:29:48.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.695+0000 7fe294928700 1 -- 172.21.15.29:0/1553959812 >> 172.21.15.29:0/1553959812 conn(0x7fe290100980 msgr2=0x7fe290102da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:29:48.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.695+0000 7fe294928700 1 -- 172.21.15.29:0/1553959812 shutdown_connections 2024-01-24T13:29:48.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.696+0000 7fe294928700 1 -- 172.21.15.29:0/1553959812 wait complete. 2024-01-24T13:29:48.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.696+0000 7fe294928700 1 Processor -- start 2024-01-24T13:29:48.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.696+0000 7fe294928700 1 -- start start 2024-01-24T13:29:48.172 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.697+0000 7fe294928700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe290104d80 0x7fe29019c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:29:48.172 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.697+0000 7fe294928700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe29019c7d0 con 0x7fe290104d80 2024-01-24T13:29:48.172 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.697+0000 7fe28e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe290104d80 0x7fe29019c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:29:48.172 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.697+0000 7fe28e59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe290104d80 0x7fe29019c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58060/0 (socket says 172.21.15.29:58060) 2024-01-24T13:29:48.172 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.697+0000 7fe28e59c700 1 -- 172.21.15.29:0/3964080341 learned_addr learned my addr 172.21.15.29:0/3964080341 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:29:48.172 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.697+0000 7fe28e59c700 1 -- 172.21.15.29:0/3964080341 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe280009b30 con 0x7fe290104d80 2024-01-24T13:29:48.172 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.697+0000 7fe28e59c700 1 --2- 172.21.15.29:0/3964080341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe290104d80 0x7fe29019c290 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe2800072a0 tx=0x7fe280004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:29:48.172 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.698+0000 7fe27f7fe700 1 -- 172.21.15.29:0/3964080341 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe28000e040 con 0x7fe290104d80 2024-01-24T13:29:48.172 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.698+0000 7fe27f7fe700 1 -- 172.21.15.29:0/3964080341 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe28000bdf0 con 0x7fe290104d80 2024-01-24T13:29:48.172 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.698+0000 7fe294928700 1 -- 172.21.15.29:0/3964080341 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe29019c9d0 con 0x7fe290104d80 2024-01-24T13:29:48.172 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.698+0000 7fe27f7fe700 1 -- 172.21.15.29:0/3964080341 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe28001c760 con 0x7fe290104d80 2024-01-24T13:29:48.172 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.698+0000 7fe294928700 1 -- 172.21.15.29:0/3964080341 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe29019ced0 con 0x7fe290104d80 2024-01-24T13:29:48.172 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.698+0000 7fe27f7fe700 1 -- 172.21.15.29:0/3964080341 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fe28001cd20 con 0x7fe290104d80 2024-01-24T13:29:48.172 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.698+0000 7fe27f7fe700 1 -- 172.21.15.29:0/3964080341 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe28001b7f0 con 0x7fe290104d80 2024-01-24T13:29:48.172 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.699+0000 7fe294928700 1 -- 172.21.15.29:0/3964080341 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe290195ae0 con 0x7fe290104d80 2024-01-24T13:29:48.173 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.702+0000 7fe27f7fe700 1 -- 172.21.15.29:0/3964080341 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fe28002a060 con 0x7fe290104d80 2024-01-24T13:29:48.173 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.779+0000 7fe294928700 1 -- 172.21.15.29:0/3964080341 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fe29002cea0 con 0x7fe290104d80 2024-01-24T13:29:48.173 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.779+0000 7fe27f7fe700 1 -- 172.21.15.29:0/3964080341 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7fe280033090 con 0x7fe290104d80 2024-01-24T13:29:48.173 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.780+0000 7fe294928700 1 -- 172.21.15.29:0/3964080341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe290104d80 msgr2=0x7fe29019c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:29:48.173 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.780+0000 7fe294928700 1 --2- 172.21.15.29:0/3964080341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe290104d80 0x7fe29019c290 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe2800072a0 tx=0x7fe280004610 comp rx=0 tx=0).stop 2024-01-24T13:29:48.173 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.780+0000 7fe294928700 1 -- 172.21.15.29:0/3964080341 shutdown_connections 2024-01-24T13:29:48.173 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.780+0000 7fe294928700 1 --2- 172.21.15.29:0/3964080341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe290104d80 0x7fe29019c290 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:29:48.173 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.780+0000 7fe294928700 1 -- 172.21.15.29:0/3964080341 >> 172.21.15.29:0/3964080341 conn(0x7fe290100980 msgr2=0x7fe290190940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:29:48.173 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.780+0000 7fe294928700 1 -- 172.21.15.29:0/3964080341 shutdown_connections 2024-01-24T13:29:48.173 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:47.780+0000 7fe294928700 1 -- 172.21.15.29:0/3964080341 wait complete. 2024-01-24T13:29:48.173 INFO:teuthology.orchestra.run.smithi029.stdout:mgr not available, waiting (3/15)... 2024-01-24T13:29:48.820 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:29:48 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:29:48.519+0000 7fd4c5e38200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-24T13:29:49.099 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:29:48 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:29:48.819+0000 7fd4c5e38200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-24T13:29:49.099 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:29:48 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:29:48.918+0000 7fd4c5e38200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-24T13:29:49.599 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:29:49 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:29:49.129+0000 7fd4c5e38200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-24T13:29:49.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:49 smithi029 ceph-mon[40651]: Activating manager daemon x 2024-01-24T13:29:49.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:49 smithi029 ceph-mon[40651]: mgrmap e2: x(active, starting, since 0.0031469s) 2024-01-24T13:29:49.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:49 smithi029 ceph-mon[40651]: from='mgr.14100 172.21.15.29:0/3717112232' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-24T13:29:49.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:49 smithi029 ceph-mon[40651]: from='mgr.14100 172.21.15.29:0/3717112232' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-24T13:29:49.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:49 smithi029 ceph-mon[40651]: from='mgr.14100 172.21.15.29:0/3717112232' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-24T13:29:49.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:49 smithi029 ceph-mon[40651]: from='mgr.14100 172.21.15.29:0/3717112232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-01-24T13:29:49.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:49 smithi029 ceph-mon[40651]: from='mgr.14100 172.21.15.29:0/3717112232' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-01-24T13:29:49.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:49 smithi029 ceph-mon[40651]: Manager daemon x is now available 2024-01-24T13:29:49.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:49 smithi029 ceph-mon[40651]: from='mgr.14100 172.21.15.29:0/3717112232' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-01-24T13:29:49.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:49 smithi029 ceph-mon[40651]: from='mgr.14100 172.21.15.29:0/3717112232' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-01-24T13:29:49.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:49 smithi029 ceph-mon[40651]: from='mgr.14100 172.21.15.29:0/3717112232' entity='mgr.x' 2024-01-24T13:29:49.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:49 smithi029 ceph-mon[40651]: from='mgr.14100 172.21.15.29:0/3717112232' entity='mgr.x' 2024-01-24T13:29:49.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:49 smithi029 ceph-mon[40651]: from='mgr.14100 172.21.15.29:0/3717112232' entity='mgr.x' 2024-01-24T13:29:51.447 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:51 smithi029 ceph-mon[40651]: mgrmap e3: x(active, since 1.00721s) 2024-01-24T13:29:52.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:52 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/3608118379' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-24T13:29:53.377 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-01-24T13:29:53.377 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-01-24T13:29:53.377 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsid": "83afcf5e-babc-11ee-95b1-87774f69a715", 2024-01-24T13:29:53.378 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "health": { 2024-01-24T13:29:53.378 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-01-24T13:29:53.378 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-01-24T13:29:53.378 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-01-24T13:29:53.378 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-01-24T13:29:53.378 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-01-24T13:29:53.378 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-01-24T13:29:53.378 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 0 2024-01-24T13:29:53.378 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-01-24T13:29:53.378 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-01-24T13:29:53.378 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "a" 2024-01-24T13:29:53.378 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-01-24T13:29:53.378 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2024-01-24T13:29:53.378 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "monmap": { 2024-01-24T13:29:53.378 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-24T13:29:53.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-01-24T13:29:53.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-01-24T13:29:53.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-01-24T13:29:53.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-01-24T13:29:53.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-24T13:29:53.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-01-24T13:29:53.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-01-24T13:29:53.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-01-24T13:29:53.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-01-24T13:29:53.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-01-24T13:29:53.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-01-24T13:29:53.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-01-24T13:29:53.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-01-24T13:29:53.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-01-24T13:29:53.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-01-24T13:29:53.380 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-01-24T13:29:53.380 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-01-24T13:29:53.380 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-01-24T13:29:53.380 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-01-24T13:29:53.380 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-01-24T13:29:53.380 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-01-24T13:29:53.380 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-01-24T13:29:53.380 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-01-24T13:29:53.380 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-24T13:29:53.380 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-01-24T13:29:53.380 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-01-24T13:29:53.380 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-01-24T13:29:53.381 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-01-24T13:29:53.381 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": true, 2024-01-24T13:29:53.381 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-01-24T13:29:53.381 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modules": [ 2024-01-24T13:29:53.381 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "iostat", 2024-01-24T13:29:53.381 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "nfs", 2024-01-24T13:29:53.381 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "restful" 2024-01-24T13:29:53.381 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-01-24T13:29:53.381 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-01-24T13:29:53.381 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-01-24T13:29:53.381 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-01-24T13:29:53.381 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-24T13:29:53.381 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modified": "2024-01-24T13:29:33.176561+0000", 2024-01-24T13:29:53.381 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-01-24T13:29:53.382 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-01-24T13:29:53.382 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-01-24T13:29:53.382 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-01-24T13:29:53.382 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:51.700+0000 7f6e6e156700 1 Processor -- start 2024-01-24T13:29:53.382 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:51.700+0000 7f6e6e156700 1 -- start start 2024-01-24T13:29:53.382 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:51.700+0000 7f6e6e156700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e68106f50 0x7f6e68109340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:29:53.382 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:51.700+0000 7f6e6e156700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e68079e50 con 0x7f6e68106f50 2024-01-24T13:29:53.382 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:51.701+0000 7f6e677fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e68106f50 0x7f6e68109340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:29:53.382 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:51.701+0000 7f6e677fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e68106f50 0x7f6e68109340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55558/0 (socket says 172.21.15.29:55558) 2024-01-24T13:29:53.382 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:51.701+0000 7f6e677fe700 1 -- 172.21.15.29:0/492401078 learned_addr learned my addr 172.21.15.29:0/492401078 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:29:53.382 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:51.701+0000 7f6e677fe700 1 -- 172.21.15.29:0/492401078 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e68109880 con 0x7f6e68106f50 2024-01-24T13:29:53.382 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:51.701+0000 7f6e677fe700 1 --2- 172.21.15.29:0/492401078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e68106f50 0x7f6e68109340 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6e580098a0 tx=0x7f6e58009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd70eb70ab22033d server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:29:53.382 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:51.701+0000 7f6e667fc700 1 -- 172.21.15.29:0/492401078 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6e58004160 con 0x7f6e68106f50 2024-01-24T13:29:53.382 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:51.701+0000 7f6e667fc700 1 -- 172.21.15.29:0/492401078 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6e580042c0 con 0x7f6e68106f50 2024-01-24T13:29:53.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:51.701+0000 7f6e667fc700 1 -- 172.21.15.29:0/492401078 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6e58011a00 con 0x7f6e68106f50 2024-01-24T13:29:53.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:51.702+0000 7f6e6e156700 1 -- 172.21.15.29:0/492401078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e68106f50 msgr2=0x7f6e68109340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:29:53.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:51.702+0000 7f6e6e156700 1 --2- 172.21.15.29:0/492401078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e68106f50 0x7f6e68109340 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6e580098a0 tx=0x7f6e58009bb0 comp rx=0 tx=0).stop 2024-01-24T13:29:53.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:51.702+0000 7f6e6e156700 1 -- 172.21.15.29:0/492401078 shutdown_connections 2024-01-24T13:29:53.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:51.702+0000 7f6e6e156700 1 --2- 172.21.15.29:0/492401078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e68106f50 0x7f6e68109340 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:29:53.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:51.702+0000 7f6e6e156700 1 -- 172.21.15.29:0/492401078 >> 172.21.15.29:0/492401078 conn(0x7f6e68100980 msgr2=0x7f6e68102da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:29:53.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:51.702+0000 7f6e6e156700 1 -- 172.21.15.29:0/492401078 shutdown_connections 2024-01-24T13:29:53.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:51.702+0000 7f6e6e156700 1 -- 172.21.15.29:0/492401078 wait complete. 2024-01-24T13:29:53.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:51.703+0000 7f6e6e156700 1 Processor -- start 2024-01-24T13:29:53.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:51.703+0000 7f6e6e156700 1 -- start start 2024-01-24T13:29:53.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:51.703+0000 7f6e6e156700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e68106f50 0x7f6e6819a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:29:53.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:51.703+0000 7f6e6e156700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6819a5d0 con 0x7f6e68106f50 2024-01-24T13:29:53.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:51.703+0000 7f6e677fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e68106f50 0x7f6e6819a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:29:53.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:51.703+0000 7f6e677fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e68106f50 0x7f6e6819a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55574/0 (socket says 172.21.15.29:55574) 2024-01-24T13:29:53.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:51.703+0000 7f6e677fe700 1 -- 172.21.15.29:0/3608118379 learned_addr learned my addr 172.21.15.29:0/3608118379 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:29:53.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:51.703+0000 7f6e677fe700 1 -- 172.21.15.29:0/3608118379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e58009550 con 0x7f6e68106f50 2024-01-24T13:29:53.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:51.704+0000 7f6e677fe700 1 --2- 172.21.15.29:0/3608118379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e68106f50 0x7f6e6819a090 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6e58006af0 tx=0x7f6e58011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:29:53.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:51.704+0000 7f6e64ff9700 1 -- 172.21.15.29:0/3608118379 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6e58003a40 con 0x7f6e68106f50 2024-01-24T13:29:53.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:51.704+0000 7f6e64ff9700 1 -- 172.21.15.29:0/3608118379 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6e58003ba0 con 0x7f6e68106f50 2024-01-24T13:29:53.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:51.704+0000 7f6e6e156700 1 -- 172.21.15.29:0/3608118379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e6819a7d0 con 0x7f6e68106f50 2024-01-24T13:29:53.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:51.704+0000 7f6e64ff9700 1 -- 172.21.15.29:0/3608118379 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6e5801a6e0 con 0x7f6e68106f50 2024-01-24T13:29:53.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:51.704+0000 7f6e6e156700 1 -- 172.21.15.29:0/3608118379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e6819ac70 con 0x7f6e68106f50 2024-01-24T13:29:53.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:51.705+0000 7f6e64ff9700 1 -- 172.21.15.29:0/3608118379 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 3) v1 ==== 45151+0+0 (secure 0 0 0) 0x7f6e5801a950 con 0x7f6e68106f50 2024-01-24T13:29:53.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:51.705+0000 7f6e6e156700 1 -- 172.21.15.29:0/3608118379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e6810a620 con 0x7f6e68106f50 2024-01-24T13:29:53.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:51.705+0000 7f6e64ff9700 1 --2- 172.21.15.29:0/3608118379 >> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] conn(0x7f6e50038180 0x7f6e5003a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:29:53.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:51.705+0000 7f6e64ff9700 1 -- 172.21.15.29:0/3608118379 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6e5804c140 con 0x7f6e68106f50 2024-01-24T13:29:53.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:51.705+0000 7f6e66ffd700 1 --2- 172.21.15.29:0/3608118379 >> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] conn(0x7f6e50038180 0x7f6e5003a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:29:53.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:51.706+0000 7f6e66ffd700 1 --2- 172.21.15.29:0/3608118379 >> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] conn(0x7f6e50038180 0x7f6e5003a640 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c006fd0 tx=0x7f6e5c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:29:53.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:51.710+0000 7f6e64ff9700 1 -- 172.21.15.29:0/3608118379 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f6e5801ac00 con 0x7f6e68106f50 2024-01-24T13:29:53.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:52.001+0000 7f6e6e156700 1 -- 172.21.15.29:0/3608118379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f6e6810a840 con 0x7f6e68106f50 2024-01-24T13:29:53.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:52.002+0000 7f6e64ff9700 1 -- 172.21.15.29:0/3608118379 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f6e58018350 con 0x7f6e68106f50 2024-01-24T13:29:53.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:52.004+0000 7f6e6e156700 1 -- 172.21.15.29:0/3608118379 >> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] conn(0x7f6e50038180 msgr2=0x7f6e5003a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:29:53.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:52.004+0000 7f6e6e156700 1 --2- 172.21.15.29:0/3608118379 >> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] conn(0x7f6e50038180 0x7f6e5003a640 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c006fd0 tx=0x7f6e5c006e40 comp rx=0 tx=0).stop 2024-01-24T13:29:53.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:52.004+0000 7f6e6e156700 1 -- 172.21.15.29:0/3608118379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e68106f50 msgr2=0x7f6e6819a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:29:53.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:52.004+0000 7f6e6e156700 1 --2- 172.21.15.29:0/3608118379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e68106f50 0x7f6e6819a090 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6e58006af0 tx=0x7f6e58011ad0 comp rx=0 tx=0).stop 2024-01-24T13:29:53.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:52.004+0000 7f6e6e156700 1 -- 172.21.15.29:0/3608118379 shutdown_connections 2024-01-24T13:29:53.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:52.004+0000 7f6e6e156700 1 --2- 172.21.15.29:0/3608118379 >> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] conn(0x7f6e50038180 0x7f6e5003a640 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:29:53.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:52.004+0000 7f6e6e156700 1 --2- 172.21.15.29:0/3608118379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e68106f50 0x7f6e6819a090 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:29:53.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:52.004+0000 7f6e6e156700 1 -- 172.21.15.29:0/3608118379 >> 172.21.15.29:0/3608118379 conn(0x7f6e68100980 msgr2=0x7f6e681016c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:29:53.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:52.004+0000 7f6e6e156700 1 -- 172.21.15.29:0/3608118379 shutdown_connections 2024-01-24T13:29:53.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:52.004+0000 7f6e6e156700 1 -- 172.21.15.29:0/3608118379 wait complete. 2024-01-24T13:29:53.385 INFO:teuthology.orchestra.run.smithi029.stdout:mgr is available 2024-01-24T13:29:53.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:53 smithi029 ceph-mon[40651]: mgrmap e4: x(active, since 3s) 2024-01-24T13:29:56.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:29:55 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/2189351457' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-24T13:29:58.561 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-01-24T13:29:58.568 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [global] 2024-01-24T13:29:58.568 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout fsid = 83afcf5e-babc-11ee-95b1-87774f69a715 2024-01-24T13:29:58.568 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-01-24T13:29:58.568 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-01-24T13:29:58.568 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-01-24T13:29:58.568 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-01-24T13:29:58.568 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-01-24T13:29:58.568 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [mgr] 2024-01-24T13:29:58.568 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-01-24T13:29:58.568 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-01-24T13:29:58.568 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [osd] 2024-01-24T13:29:58.568 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-01-24T13:29:58.569 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-01-24T13:29:58.569 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-01-24T13:29:58.569 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-01-24T13:29:58.569 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-01-24T13:29:58.569 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.643+0000 7fca960f6700 1 Processor -- start 2024-01-24T13:29:58.569 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.643+0000 7fca960f6700 1 -- start start 2024-01-24T13:29:58.569 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.643+0000 7fca960f6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca90106f10 0x7fca90109300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:29:58.569 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.644+0000 7fca960f6700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca90079e90 con 0x7fca90106f10 2024-01-24T13:29:58.569 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.644+0000 7fca8f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca90106f10 0x7fca90109300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:29:58.569 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.644+0000 7fca8f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca90106f10 0x7fca90109300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55586/0 (socket says 172.21.15.29:55586) 2024-01-24T13:29:58.569 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.644+0000 7fca8f7fe700 1 -- 172.21.15.29:0/813288192 learned_addr learned my addr 172.21.15.29:0/813288192 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:29:58.569 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.644+0000 7fca8f7fe700 1 -- 172.21.15.29:0/813288192 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca90109840 con 0x7fca90106f10 2024-01-24T13:29:58.569 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.644+0000 7fca8f7fe700 1 --2- 172.21.15.29:0/813288192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca90106f10 0x7fca90109300 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fca800098a0 tx=0x7fca80009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96c349fa8350f819 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:29:58.569 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.644+0000 7fca8e7fc700 1 -- 172.21.15.29:0/813288192 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fca80004160 con 0x7fca90106f10 2024-01-24T13:29:58.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.644+0000 7fca8e7fc700 1 -- 172.21.15.29:0/813288192 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fca800042c0 con 0x7fca90106f10 2024-01-24T13:29:58.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.644+0000 7fca8e7fc700 1 -- 172.21.15.29:0/813288192 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fca80011a00 con 0x7fca90106f10 2024-01-24T13:29:58.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.645+0000 7fca960f6700 1 -- 172.21.15.29:0/813288192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca90106f10 msgr2=0x7fca90109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:29:58.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.645+0000 7fca960f6700 1 --2- 172.21.15.29:0/813288192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca90106f10 0x7fca90109300 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fca800098a0 tx=0x7fca80009bb0 comp rx=0 tx=0).stop 2024-01-24T13:29:58.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.645+0000 7fca960f6700 1 -- 172.21.15.29:0/813288192 shutdown_connections 2024-01-24T13:29:58.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.645+0000 7fca960f6700 1 --2- 172.21.15.29:0/813288192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca90106f10 0x7fca90109300 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:29:58.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.645+0000 7fca960f6700 1 -- 172.21.15.29:0/813288192 >> 172.21.15.29:0/813288192 conn(0x7fca90100980 msgr2=0x7fca90102da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:29:58.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.645+0000 7fca960f6700 1 -- 172.21.15.29:0/813288192 shutdown_connections 2024-01-24T13:29:58.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.645+0000 7fca960f6700 1 -- 172.21.15.29:0/813288192 wait complete. 2024-01-24T13:29:58.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.646+0000 7fca960f6700 1 Processor -- start 2024-01-24T13:29:58.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.646+0000 7fca960f6700 1 -- start start 2024-01-24T13:29:58.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.646+0000 7fca960f6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca90106f10 0x7fca90106870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:29:58.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.646+0000 7fca960f6700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca90106db0 con 0x7fca90106f10 2024-01-24T13:29:58.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.646+0000 7fca8f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca90106f10 0x7fca90106870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:29:58.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.646+0000 7fca8f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca90106f10 0x7fca90106870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55592/0 (socket says 172.21.15.29:55592) 2024-01-24T13:29:58.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.646+0000 7fca8f7fe700 1 -- 172.21.15.29:0/2189351457 learned_addr learned my addr 172.21.15.29:0/2189351457 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:29:58.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.647+0000 7fca8f7fe700 1 -- 172.21.15.29:0/2189351457 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca80009550 con 0x7fca90106f10 2024-01-24T13:29:58.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.647+0000 7fca8f7fe700 1 --2- 172.21.15.29:0/2189351457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca90106f10 0x7fca90106870 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fca80009fb0 tx=0x7fca80011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:29:58.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.647+0000 7fca8cff9700 1 -- 172.21.15.29:0/2189351457 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fca80003930 con 0x7fca90106f10 2024-01-24T13:29:58.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.647+0000 7fca8cff9700 1 -- 172.21.15.29:0/2189351457 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fca80003a90 con 0x7fca90106f10 2024-01-24T13:29:58.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.647+0000 7fca960f6700 1 -- 172.21.15.29:0/2189351457 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca90104ec0 con 0x7fca90106f10 2024-01-24T13:29:58.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.647+0000 7fca8cff9700 1 -- 172.21.15.29:0/2189351457 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fca80019610 con 0x7fca90106f10 2024-01-24T13:29:58.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.647+0000 7fca960f6700 1 -- 172.21.15.29:0/2189351457 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca90105360 con 0x7fca90106f10 2024-01-24T13:29:58.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.648+0000 7fca8cff9700 1 -- 172.21.15.29:0/2189351457 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 4) v1 ==== 45257+0+0 (secure 0 0 0) 0x7fca80019880 con 0x7fca90106f10 2024-01-24T13:29:58.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.648+0000 7fca8cff9700 1 --2- 172.21.15.29:0/2189351457 >> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] conn(0x7fca780382b0 0x7fca7803a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:29:58.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.648+0000 7fca8cff9700 1 -- 172.21.15.29:0/2189351457 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fca8004bfb0 con 0x7fca90106f10 2024-01-24T13:29:58.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.648+0000 7fca8effd700 1 --2- 172.21.15.29:0/2189351457 >> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] conn(0x7fca780382b0 0x7fca7803a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:29:58.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.648+0000 7fca960f6700 1 -- 172.21.15.29:0/2189351457 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca9010a640 con 0x7fca90106f10 2024-01-24T13:29:58.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.649+0000 7fca8effd700 1 --2- 172.21.15.29:0/2189351457 >> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] conn(0x7fca780382b0 0x7fca7803a770 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fca84006fd0 tx=0x7fca84006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:29:58.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.653+0000 7fca8cff9700 1 -- 172.21.15.29:0/2189351457 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fca80019b30 con 0x7fca90106f10 2024-01-24T13:29:58.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.861+0000 7fca960f6700 1 -- 172.21.15.29:0/2189351457 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fca9010a800 con 0x7fca90106f10 2024-01-24T13:29:58.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.862+0000 7fca8cff9700 1 -- 172.21.15.29:0/2189351457 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7fca80018b40 con 0x7fca90106f10 2024-01-24T13:29:58.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.864+0000 7fca960f6700 1 -- 172.21.15.29:0/2189351457 >> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] conn(0x7fca780382b0 msgr2=0x7fca7803a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:29:58.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.864+0000 7fca960f6700 1 --2- 172.21.15.29:0/2189351457 >> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] conn(0x7fca780382b0 0x7fca7803a770 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fca84006fd0 tx=0x7fca84006e40 comp rx=0 tx=0).stop 2024-01-24T13:29:58.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.864+0000 7fca960f6700 1 -- 172.21.15.29:0/2189351457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca90106f10 msgr2=0x7fca90106870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:29:58.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.864+0000 7fca960f6700 1 --2- 172.21.15.29:0/2189351457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca90106f10 0x7fca90106870 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fca80009fb0 tx=0x7fca80011ad0 comp rx=0 tx=0).stop 2024-01-24T13:29:58.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.864+0000 7fca960f6700 1 -- 172.21.15.29:0/2189351457 shutdown_connections 2024-01-24T13:29:58.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.864+0000 7fca960f6700 1 --2- 172.21.15.29:0/2189351457 >> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] conn(0x7fca780382b0 0x7fca7803a770 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:29:58.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.864+0000 7fca960f6700 1 --2- 172.21.15.29:0/2189351457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca90106f10 0x7fca90106870 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:29:58.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.864+0000 7fca960f6700 1 -- 172.21.15.29:0/2189351457 >> 172.21.15.29:0/2189351457 conn(0x7fca90100980 msgr2=0x7fca901016d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:29:58.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.864+0000 7fca960f6700 1 -- 172.21.15.29:0/2189351457 shutdown_connections 2024-01-24T13:29:58.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:29:55.864+0000 7fca960f6700 1 -- 172.21.15.29:0/2189351457 wait complete. 2024-01-24T13:29:58.572 INFO:teuthology.orchestra.run.smithi029.stdout:Enabling cephadm module... 2024-01-24T13:30:01.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:01 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/3964879757' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-01-24T13:30:01.600 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:01 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: ignoring --setuser ceph since I am not root 2024-01-24T13:30:01.600 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:01 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: ignoring --setgroup ceph since I am not root 2024-01-24T13:30:01.938 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:01 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:01.633+0000 7f6d625a9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-24T13:30:01.938 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:01 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:01.738+0000 7f6d625a9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-24T13:30:01.938 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:01 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:01.836+0000 7f6d625a9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-24T13:30:01.938 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:01 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:01.937+0000 7f6d625a9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-24T13:30:02.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:02 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/3964879757' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-01-24T13:30:02.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:02 smithi029 ceph-mon[40651]: mgrmap e5: x(active, since 12s) 2024-01-24T13:30:02.922 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:02 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:02.654+0000 7f6d625a9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-24T13:30:03.205 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:00.814+0000 7f403800d700 1 Processor -- start 2024-01-24T13:30:03.206 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:00.815+0000 7f403800d700 1 -- start start 2024-01-24T13:30:03.206 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:00.815+0000 7f403800d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4030106f50 0x7f4030109340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:03.206 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:00.815+0000 7f403800d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4030079e30 con 0x7f4030106f50 2024-01-24T13:30:03.206 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:00.815+0000 7f4035da9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4030106f50 0x7f4030109340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:03.206 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:00.815+0000 7f4035da9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4030106f50 0x7f4030109340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55598/0 (socket says 172.21.15.29:55598) 2024-01-24T13:30:03.206 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:00.815+0000 7f4035da9700 1 -- 172.21.15.29:0/4022755531 learned_addr learned my addr 172.21.15.29:0/4022755531 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:03.206 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:00.816+0000 7f4035da9700 1 -- 172.21.15.29:0/4022755531 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4030109880 con 0x7f4030106f50 2024-01-24T13:30:03.206 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:00.816+0000 7f4035da9700 1 --2- 172.21.15.29:0/4022755531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4030106f50 0x7f4030109340 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f402c0098a0 tx=0x7f402c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d87164a49750163 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:03.206 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:00.816+0000 7f4034da7700 1 -- 172.21.15.29:0/4022755531 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f402c004160 con 0x7f4030106f50 2024-01-24T13:30:03.206 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:00.816+0000 7f4034da7700 1 -- 172.21.15.29:0/4022755531 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f402c0042c0 con 0x7f4030106f50 2024-01-24T13:30:03.206 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:00.816+0000 7f4034da7700 1 -- 172.21.15.29:0/4022755531 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f402c011a00 con 0x7f4030106f50 2024-01-24T13:30:03.207 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:00.817+0000 7f403800d700 1 -- 172.21.15.29:0/4022755531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4030106f50 msgr2=0x7f4030109340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:03.207 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:00.817+0000 7f403800d700 1 --2- 172.21.15.29:0/4022755531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4030106f50 0x7f4030109340 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f402c0098a0 tx=0x7f402c009bb0 comp rx=0 tx=0).stop 2024-01-24T13:30:03.207 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:00.817+0000 7f403800d700 1 -- 172.21.15.29:0/4022755531 shutdown_connections 2024-01-24T13:30:03.207 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:00.817+0000 7f403800d700 1 --2- 172.21.15.29:0/4022755531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4030106f50 0x7f4030109340 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:03.207 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:00.817+0000 7f403800d700 1 -- 172.21.15.29:0/4022755531 >> 172.21.15.29:0/4022755531 conn(0x7f4030100940 msgr2=0x7f4030102d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:03.207 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:00.817+0000 7f403800d700 1 -- 172.21.15.29:0/4022755531 shutdown_connections 2024-01-24T13:30:03.207 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:00.817+0000 7f403800d700 1 -- 172.21.15.29:0/4022755531 wait complete. 2024-01-24T13:30:03.207 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:00.818+0000 7f403800d700 1 Processor -- start 2024-01-24T13:30:03.207 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:00.818+0000 7f403800d700 1 -- start start 2024-01-24T13:30:03.207 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:00.818+0000 7f403800d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4030106f50 0x7f403019a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:03.207 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:00.818+0000 7f403800d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403019a640 con 0x7f4030106f50 2024-01-24T13:30:03.207 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:00.818+0000 7f4035da9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4030106f50 0x7f403019a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:03.207 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:00.818+0000 7f4035da9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4030106f50 0x7f403019a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55600/0 (socket says 172.21.15.29:55600) 2024-01-24T13:30:03.207 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:00.818+0000 7f4035da9700 1 -- 172.21.15.29:0/3964879757 learned_addr learned my addr 172.21.15.29:0/3964879757 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:03.208 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:00.818+0000 7f4035da9700 1 -- 172.21.15.29:0/3964879757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f402c009550 con 0x7f4030106f50 2024-01-24T13:30:03.208 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:00.819+0000 7f4035da9700 1 --2- 172.21.15.29:0/3964879757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4030106f50 0x7f403019a100 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f402c009130 tx=0x7f402c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:03.208 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:00.819+0000 7f4026ffd700 1 -- 172.21.15.29:0/3964879757 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f402c003b70 con 0x7f4030106f50 2024-01-24T13:30:03.208 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:00.819+0000 7f4026ffd700 1 -- 172.21.15.29:0/3964879757 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f402c003cd0 con 0x7f4030106f50 2024-01-24T13:30:03.208 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:00.819+0000 7f4026ffd700 1 -- 172.21.15.29:0/3964879757 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f402c0197d0 con 0x7f4030106f50 2024-01-24T13:30:03.208 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:00.819+0000 7f403800d700 1 -- 172.21.15.29:0/3964879757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f403019a840 con 0x7f4030106f50 2024-01-24T13:30:03.208 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:00.819+0000 7f403800d700 1 -- 172.21.15.29:0/3964879757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f403019ace0 con 0x7f4030106f50 2024-01-24T13:30:03.208 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:00.820+0000 7f403800d700 1 -- 172.21.15.29:0/3964879757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f403010a620 con 0x7f4030106f50 2024-01-24T13:30:03.208 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:00.820+0000 7f4026ffd700 1 -- 172.21.15.29:0/3964879757 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 4) v1 ==== 45257+0+0 (secure 0 0 0) 0x7f402c01abb0 con 0x7f4030106f50 2024-01-24T13:30:03.208 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:00.821+0000 7f4026ffd700 1 --2- 172.21.15.29:0/3964879757 >> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] conn(0x7f401c038300 0x7f401c03a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:03.208 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:00.821+0000 7f4026ffd700 1 -- 172.21.15.29:0/3964879757 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f402c04c2c0 con 0x7f4030106f50 2024-01-24T13:30:03.208 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:00.821+0000 7f40355a8700 1 --2- 172.21.15.29:0/3964879757 >> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] conn(0x7f401c038300 0x7f401c03a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:03.208 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:00.821+0000 7f40355a8700 1 --2- 172.21.15.29:0/3964879757 >> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] conn(0x7f401c038300 0x7f401c03a7c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4020006fd0 tx=0x7f4020006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:03.208 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:00.825+0000 7f4026ffd700 1 -- 172.21.15.29:0/3964879757 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f402c01a430 con 0x7f4030106f50 2024-01-24T13:30:03.209 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:01.090+0000 7f403800d700 1 -- 172.21.15.29:0/3964879757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f403019dac0 con 0x7f4030106f50 2024-01-24T13:30:03.209 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:01.144+0000 7f4026ffd700 1 -- 172.21.15.29:0/3964879757 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 5) v1 ==== 45268+0+0 (secure 0 0 0) 0x7f402c02bd40 con 0x7f4030106f50 2024-01-24T13:30:03.209 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:01.145+0000 7f4026ffd700 1 -- 172.21.15.29:0/3964879757 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f402c02dca0 con 0x7f4030106f50 2024-01-24T13:30:03.209 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:01.149+0000 7f403800d700 1 -- 172.21.15.29:0/3964879757 >> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] conn(0x7f401c038300 msgr2=0x7f401c03a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:03.209 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:01.149+0000 7f403800d700 1 --2- 172.21.15.29:0/3964879757 >> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] conn(0x7f401c038300 0x7f401c03a7c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4020006fd0 tx=0x7f4020006e40 comp rx=0 tx=0).stop 2024-01-24T13:30:03.209 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:01.149+0000 7f403800d700 1 -- 172.21.15.29:0/3964879757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4030106f50 msgr2=0x7f403019a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:03.209 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:01.149+0000 7f403800d700 1 --2- 172.21.15.29:0/3964879757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4030106f50 0x7f403019a100 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f402c009130 tx=0x7f402c011ad0 comp rx=0 tx=0).stop 2024-01-24T13:30:03.209 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:02 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:02.921+0000 7f6d625a9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-24T13:30:03.209 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:03 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:03.016+0000 7f6d625a9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-24T13:30:03.209 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:03 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:03.139+0000 7f6d625a9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-24T13:30:03.210 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:01.149+0000 7f403800d700 1 -- 172.21.15.29:0/3964879757 shutdown_connections 2024-01-24T13:30:03.210 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:01.149+0000 7f403800d700 1 --2- 172.21.15.29:0/3964879757 >> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] conn(0x7f401c038300 0x7f401c03a7c0 secure :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4020006fd0 tx=0x7f4020006e40 comp rx=0 tx=0).stop 2024-01-24T13:30:03.210 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:01.149+0000 7f403800d700 1 --2- 172.21.15.29:0/3964879757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4030106f50 0x7f403019a100 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:03.210 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:01.149+0000 7f403800d700 1 -- 172.21.15.29:0/3964879757 >> 172.21.15.29:0/3964879757 conn(0x7f4030100940 msgr2=0x7f40301016e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:03.210 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:01.150+0000 7f403800d700 1 -- 172.21.15.29:0/3964879757 shutdown_connections 2024-01-24T13:30:03.211 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:01.150+0000 7f403800d700 1 -- 172.21.15.29:0/3964879757 wait complete. 2024-01-24T13:30:03.495 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:03 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:03.343+0000 7f6d625a9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-24T13:30:03.495 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:03 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:03.494+0000 7f6d625a9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-24T13:30:03.849 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:03 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:03.617+0000 7f6d625a9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-24T13:30:04.153 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:04 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:04.050+0000 7f6d625a9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-24T13:30:04.153 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:03 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/3870203642' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-24T13:30:04.172 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-01-24T13:30:04.173 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-01-24T13:30:04.174 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": true, 2024-01-24T13:30:04.174 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-01-24T13:30:04.174 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-01-24T13:30:04.174 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-01-24T13:30:04.174 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.566+0000 7f2afda4a700 1 Processor -- start 2024-01-24T13:30:04.174 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.566+0000 7f2afda4a700 1 -- start start 2024-01-24T13:30:04.174 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.566+0000 7f2afda4a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2af81053a0 0x7f2af81057c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:04.174 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.566+0000 7f2afda4a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2af8105d90 con 0x7f2af81053a0 2024-01-24T13:30:04.174 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.567+0000 7f2af6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2af81053a0 0x7f2af81057c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:04.174 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.567+0000 7f2af6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2af81053a0 0x7f2af81057c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46776/0 (socket says 172.21.15.29:46776) 2024-01-24T13:30:04.174 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.567+0000 7f2af6ffd700 1 -- 172.21.15.29:0/3876474605 learned_addr learned my addr 172.21.15.29:0/3876474605 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:04.174 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.567+0000 7f2af6ffd700 1 -- 172.21.15.29:0/3876474605 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af81065a0 con 0x7f2af81053a0 2024-01-24T13:30:04.174 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.567+0000 7f2af6ffd700 1 --2- 172.21.15.29:0/3876474605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2af81053a0 0x7f2af81057c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2ae80098a0 tx=0x7f2ae8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b815de0ce6f5a0c6 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:04.174 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.568+0000 7f2af5ffb700 1 -- 172.21.15.29:0/3876474605 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2ae8004160 con 0x7f2af81053a0 2024-01-24T13:30:04.179 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.568+0000 7f2af5ffb700 1 -- 172.21.15.29:0/3876474605 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2ae80042c0 con 0x7f2af81053a0 2024-01-24T13:30:04.179 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.568+0000 7f2af5ffb700 1 -- 172.21.15.29:0/3876474605 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2ae8011a00 con 0x7f2af81053a0 2024-01-24T13:30:04.179 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.569+0000 7f2afda4a700 1 -- 172.21.15.29:0/3876474605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2af81053a0 msgr2=0x7f2af81057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:04.179 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.569+0000 7f2afda4a700 1 --2- 172.21.15.29:0/3876474605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2af81053a0 0x7f2af81057c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2ae80098a0 tx=0x7f2ae8009bb0 comp rx=0 tx=0).stop 2024-01-24T13:30:04.179 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.572+0000 7f2afda4a700 1 -- 172.21.15.29:0/3876474605 shutdown_connections 2024-01-24T13:30:04.179 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.572+0000 7f2afda4a700 1 --2- 172.21.15.29:0/3876474605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2af81053a0 0x7f2af81057c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:04.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.572+0000 7f2afda4a700 1 -- 172.21.15.29:0/3876474605 >> 172.21.15.29:0/3876474605 conn(0x7f2af8100980 msgr2=0x7f2af8102da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:04.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.572+0000 7f2afda4a700 1 -- 172.21.15.29:0/3876474605 shutdown_connections 2024-01-24T13:30:04.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.572+0000 7f2afda4a700 1 -- 172.21.15.29:0/3876474605 wait complete. 2024-01-24T13:30:04.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.573+0000 7f2afda4a700 1 Processor -- start 2024-01-24T13:30:04.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.573+0000 7f2afda4a700 1 -- start start 2024-01-24T13:30:04.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.573+0000 7f2afda4a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2af81053a0 0x7f2af819c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:04.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.573+0000 7f2afda4a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2af819c840 con 0x7f2af81053a0 2024-01-24T13:30:04.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.573+0000 7f2af6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2af81053a0 0x7f2af819c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:04.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.573+0000 7f2af6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2af81053a0 0x7f2af819c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46784/0 (socket says 172.21.15.29:46784) 2024-01-24T13:30:04.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.573+0000 7f2af6ffd700 1 -- 172.21.15.29:0/3870203642 learned_addr learned my addr 172.21.15.29:0/3870203642 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:04.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.574+0000 7f2af6ffd700 1 -- 172.21.15.29:0/3870203642 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae8009550 con 0x7f2af81053a0 2024-01-24T13:30:04.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.574+0000 7f2af6ffd700 1 --2- 172.21.15.29:0/3870203642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2af81053a0 0x7f2af819c300 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2ae8009fb0 tx=0x7f2ae8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:04.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.574+0000 7f2afca48700 1 -- 172.21.15.29:0/3870203642 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2ae8003930 con 0x7f2af81053a0 2024-01-24T13:30:04.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.574+0000 7f2afca48700 1 -- 172.21.15.29:0/3870203642 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2ae8003a90 con 0x7f2af81053a0 2024-01-24T13:30:04.181 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.574+0000 7f2afda4a700 1 -- 172.21.15.29:0/3870203642 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2af819ca40 con 0x7f2af81053a0 2024-01-24T13:30:04.181 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.574+0000 7f2afca48700 1 -- 172.21.15.29:0/3870203642 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2ae80195e0 con 0x7f2af81053a0 2024-01-24T13:30:04.181 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.574+0000 7f2afda4a700 1 -- 172.21.15.29:0/3870203642 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2af819cee0 con 0x7f2af81053a0 2024-01-24T13:30:04.181 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.575+0000 7f2afca48700 1 -- 172.21.15.29:0/3870203642 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 45268+0+0 (secure 0 0 0) 0x7f2ae8019850 con 0x7f2af81053a0 2024-01-24T13:30:04.181 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.576+0000 7f2ade7fc700 1 -- 172.21.15.29:0/3870203642 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2af8062370 con 0x7f2af81053a0 2024-01-24T13:30:04.181 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.576+0000 7f2afca48700 1 --2- 172.21.15.29:0/3870203642 >> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] conn(0x7f2ae00382f0 0x7f2ae003a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:04.181 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.576+0000 7f2afca48700 1 -- 172.21.15.29:0/3870203642 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2ae804beb0 con 0x7f2af81053a0 2024-01-24T13:30:04.181 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.576+0000 7f2af67fc700 1 -- 172.21.15.29:0/3870203642 >> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] conn(0x7f2ae00382f0 msgr2=0x7f2ae003a7b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/835547951 2024-01-24T13:30:04.181 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.576+0000 7f2af67fc700 1 --2- 172.21.15.29:0/3870203642 >> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] conn(0x7f2ae00382f0 0x7f2ae003a7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-24T13:30:04.181 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.581+0000 7f2afca48700 1 -- 172.21.15.29:0/3870203642 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f2ae802b340 con 0x7f2af81053a0 2024-01-24T13:30:04.181 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.777+0000 7f2af67fc700 1 -- 172.21.15.29:0/3870203642 >> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] conn(0x7f2ae00382f0 msgr2=0x7f2ae003a7b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/835547951 2024-01-24T13:30:04.181 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.777+0000 7f2af67fc700 1 --2- 172.21.15.29:0/3870203642 >> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] conn(0x7f2ae00382f0 0x7f2ae003a7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-24T13:30:04.181 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.851+0000 7f2ade7fc700 1 -- 172.21.15.29:0/3870203642 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f2af8078fb0 con 0x7f2af81053a0 2024-01-24T13:30:04.181 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.852+0000 7f2afca48700 1 -- 172.21.15.29:0/3870203642 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f2ae8003c00 con 0x7f2af81053a0 2024-01-24T13:30:04.182 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.854+0000 7f2ade7fc700 1 -- 172.21.15.29:0/3870203642 >> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] conn(0x7f2ae00382f0 msgr2=0x7f2ae003a7b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-24T13:30:04.182 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.854+0000 7f2ade7fc700 1 --2- 172.21.15.29:0/3870203642 >> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] conn(0x7f2ae00382f0 0x7f2ae003a7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:04.182 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.854+0000 7f2ade7fc700 1 -- 172.21.15.29:0/3870203642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2af81053a0 msgr2=0x7f2af819c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:04.182 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.854+0000 7f2ade7fc700 1 --2- 172.21.15.29:0/3870203642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2af81053a0 0x7f2af819c300 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2ae8009fb0 tx=0x7f2ae8011ad0 comp rx=0 tx=0).stop 2024-01-24T13:30:04.182 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.854+0000 7f2ade7fc700 1 -- 172.21.15.29:0/3870203642 shutdown_connections 2024-01-24T13:30:04.182 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.854+0000 7f2ade7fc700 1 --2- 172.21.15.29:0/3870203642 >> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] conn(0x7f2ae00382f0 0x7f2ae003a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:04.182 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.854+0000 7f2ade7fc700 1 --2- 172.21.15.29:0/3870203642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2af81053a0 0x7f2af819c300 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:04.182 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.854+0000 7f2ade7fc700 1 -- 172.21.15.29:0/3870203642 >> 172.21.15.29:0/3870203642 conn(0x7f2af8100980 msgr2=0x7f2af818ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:04.182 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.854+0000 7f2ade7fc700 1 -- 172.21.15.29:0/3870203642 shutdown_connections 2024-01-24T13:30:04.182 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:03.854+0000 7f2ade7fc700 1 -- 172.21.15.29:0/3870203642 wait complete. 2024-01-24T13:30:04.182 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for the mgr to restart... 2024-01-24T13:30:04.182 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mgr epoch 5... 2024-01-24T13:30:04.599 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:04 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:04.371+0000 7f6d625a9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-24T13:30:05.099 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:04 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:04.707+0000 7f6d625a9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-24T13:30:05.099 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:04 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:04.813+0000 7f6d625a9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-24T13:30:05.953 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:05 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:05.671+0000 7f6d625a9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-24T13:30:05.953 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:05 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:05.778+0000 7f6d625a9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-24T13:30:05.953 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:05 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:05.950+0000 7f6d625a9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-24T13:30:07.095 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:06 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:06.670+0000 7f6d625a9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-24T13:30:07.095 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:06 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:06.875+0000 7f6d625a9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-24T13:30:07.095 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:07 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:07.094+0000 7f6d625a9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-24T13:30:08.415 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:08 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:08.138+0000 7f6d625a9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-24T13:30:08.761 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:08 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:08.443+0000 7f6d625a9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-24T13:30:08.762 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:08 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:08.544+0000 7f6d625a9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-24T13:30:09.099 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:08 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:08.760+0000 7f6d625a9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-24T13:30:09.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:08 smithi029 ceph-mon[40651]: Active manager daemon x restarted 2024-01-24T13:30:09.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:08 smithi029 ceph-mon[40651]: Activating manager daemon x 2024-01-24T13:30:09.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:08 smithi029 ceph-mon[40651]: osdmap e2: 0 total, 0 up, 0 in 2024-01-24T13:30:09.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:08 smithi029 ceph-mon[40651]: mgrmap e6: x(active, starting, since 0.00357575s) 2024-01-24T13:30:09.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:08 smithi029 ceph-mon[40651]: from='mgr.14120 172.21.15.29:0/656713355' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-01-24T13:30:09.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:08 smithi029 ceph-mon[40651]: from='mgr.14120 172.21.15.29:0/656713355' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-01-24T13:30:09.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:08 smithi029 ceph-mon[40651]: from='mgr.14120 172.21.15.29:0/656713355' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-24T13:30:09.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:08 smithi029 ceph-mon[40651]: from='mgr.14120 172.21.15.29:0/656713355' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-24T13:30:09.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:08 smithi029 ceph-mon[40651]: from='mgr.14120 172.21.15.29:0/656713355' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-24T13:30:09.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:08 smithi029 ceph-mon[40651]: Manager daemon x is now available 2024-01-24T13:30:09.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:08 smithi029 ceph-mon[40651]: from='mgr.14120 172.21.15.29:0/656713355' entity='mgr.x' 2024-01-24T13:30:09.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:08 smithi029 ceph-mon[40651]: from='mgr.14120 172.21.15.29:0/656713355' entity='mgr.x' 2024-01-24T13:30:09.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:08 smithi029 ceph-mon[40651]: from='mgr.14120 172.21.15.29:0/656713355' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:30:10.093 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:09 smithi029 ceph-mon[40651]: Found migration_current of "None". Setting to last migration. 2024-01-24T13:30:10.094 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:09 smithi029 ceph-mon[40651]: from='mgr.14120 172.21.15.29:0/656713355' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:30:10.094 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:09 smithi029 ceph-mon[40651]: from='mgr.14120 172.21.15.29:0/656713355' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:30:10.094 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:09 smithi029 ceph-mon[40651]: from='mgr.14120 172.21.15.29:0/656713355' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-01-24T13:30:10.094 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:09 smithi029 ceph-mon[40651]: from='mgr.14120 172.21.15.29:0/656713355' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-01-24T13:30:10.094 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:09 smithi029 ceph-mon[40651]: from='mgr.14120 172.21.15.29:0/656713355' entity='mgr.x' 2024-01-24T13:30:10.094 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:09 smithi029 ceph-mon[40651]: from='mgr.14120 172.21.15.29:0/656713355' entity='mgr.x' 2024-01-24T13:30:10.094 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:09 smithi029 ceph-mon[40651]: mgrmap e7: x(active, since 1.00617s) 2024-01-24T13:30:10.167 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-01-24T13:30:10.176 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-01-24T13:30:10.176 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "initialized": true 2024-01-24T13:30:10.176 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-01-24T13:30:10.176 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.556+0000 7f5d65621700 1 Processor -- start 2024-01-24T13:30:10.176 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.556+0000 7f5d65621700 1 -- start start 2024-01-24T13:30:10.177 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.556+0000 7f5d65621700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d60104790 0x7f5d60104bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:10.177 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.556+0000 7f5d65621700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d60105180 con 0x7f5d60104790 2024-01-24T13:30:10.177 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.557+0000 7f5d5effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d60104790 0x7f5d60104bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:10.177 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.557+0000 7f5d5effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d60104790 0x7f5d60104bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46800/0 (socket says 172.21.15.29:46800) 2024-01-24T13:30:10.177 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.557+0000 7f5d5effd700 1 -- 172.21.15.29:0/384502904 learned_addr learned my addr 172.21.15.29:0/384502904 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:10.177 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.557+0000 7f5d5effd700 1 -- 172.21.15.29:0/384502904 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d60107720 con 0x7f5d60104790 2024-01-24T13:30:10.177 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.557+0000 7f5d5effd700 1 --2- 172.21.15.29:0/384502904 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d60104790 0x7f5d60104bb0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5d500098a0 tx=0x7f5d50009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=675d348412d26c6d server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:10.177 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.557+0000 7f5d5dffb700 1 -- 172.21.15.29:0/384502904 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5d50004160 con 0x7f5d60104790 2024-01-24T13:30:10.177 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.558+0000 7f5d5dffb700 1 -- 172.21.15.29:0/384502904 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5d500042c0 con 0x7f5d60104790 2024-01-24T13:30:10.177 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.558+0000 7f5d5dffb700 1 -- 172.21.15.29:0/384502904 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5d50011a00 con 0x7f5d60104790 2024-01-24T13:30:10.177 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.558+0000 7f5d65621700 1 -- 172.21.15.29:0/384502904 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d60104790 msgr2=0x7f5d60104bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:10.177 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.558+0000 7f5d65621700 1 --2- 172.21.15.29:0/384502904 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d60104790 0x7f5d60104bb0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5d500098a0 tx=0x7f5d50009bb0 comp rx=0 tx=0).stop 2024-01-24T13:30:10.178 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.558+0000 7f5d65621700 1 -- 172.21.15.29:0/384502904 shutdown_connections 2024-01-24T13:30:10.178 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.558+0000 7f5d65621700 1 --2- 172.21.15.29:0/384502904 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d60104790 0x7f5d60104bb0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:10.178 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.558+0000 7f5d65621700 1 -- 172.21.15.29:0/384502904 >> 172.21.15.29:0/384502904 conn(0x7f5d60100140 msgr2=0x7f5d60102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:10.178 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.558+0000 7f5d65621700 1 -- 172.21.15.29:0/384502904 shutdown_connections 2024-01-24T13:30:10.178 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.558+0000 7f5d65621700 1 -- 172.21.15.29:0/384502904 wait complete. 2024-01-24T13:30:10.178 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.559+0000 7f5d65621700 1 Processor -- start 2024-01-24T13:30:10.178 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.560+0000 7f5d65621700 1 -- start start 2024-01-24T13:30:10.178 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.560+0000 7f5d65621700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d60104790 0x7f5d6019a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:10.178 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.560+0000 7f5d65621700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6019a640 con 0x7f5d60104790 2024-01-24T13:30:10.178 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.560+0000 7f5d5effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d60104790 0x7f5d6019a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:10.178 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.560+0000 7f5d5effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d60104790 0x7f5d6019a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46802/0 (socket says 172.21.15.29:46802) 2024-01-24T13:30:10.178 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.560+0000 7f5d5effd700 1 -- 172.21.15.29:0/179038770 learned_addr learned my addr 172.21.15.29:0/179038770 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:10.179 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.560+0000 7f5d5effd700 1 -- 172.21.15.29:0/179038770 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d50009550 con 0x7f5d60104790 2024-01-24T13:30:10.179 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.561+0000 7f5d5effd700 1 --2- 172.21.15.29:0/179038770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d60104790 0x7f5d6019a100 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5d50006af0 tx=0x7f5d50011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:10.179 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.561+0000 7f5d47fff700 1 -- 172.21.15.29:0/179038770 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5d50003a40 con 0x7f5d60104790 2024-01-24T13:30:10.179 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.561+0000 7f5d47fff700 1 -- 172.21.15.29:0/179038770 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5d50003ba0 con 0x7f5d60104790 2024-01-24T13:30:10.179 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.561+0000 7f5d65621700 1 -- 172.21.15.29:0/179038770 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d6019a840 con 0x7f5d60104790 2024-01-24T13:30:10.179 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.561+0000 7f5d65621700 1 -- 172.21.15.29:0/179038770 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d6019ace0 con 0x7f5d60104790 2024-01-24T13:30:10.179 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.561+0000 7f5d47fff700 1 -- 172.21.15.29:0/179038770 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5d5001a700 con 0x7f5d60104790 2024-01-24T13:30:10.179 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.562+0000 7f5d47fff700 1 -- 172.21.15.29:0/179038770 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 45268+0+0 (secure 0 0 0) 0x7f5d5001a930 con 0x7f5d60104790 2024-01-24T13:30:10.179 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.562+0000 7f5d47fff700 1 --2- 172.21.15.29:0/179038770 >> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] conn(0x7f5d48038350 0x7f5d4803a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:10.179 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.562+0000 7f5d47fff700 1 -- 172.21.15.29:0/179038770 --> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5d4803af20 con 0x7f5d48038350 2024-01-24T13:30:10.179 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.562+0000 7f5d47fff700 1 -- 172.21.15.29:0/179038770 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f5d5004c040 con 0x7f5d60104790 2024-01-24T13:30:10.179 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.562+0000 7f5d5e7fc700 1 -- 172.21.15.29:0/179038770 >> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] conn(0x7f5d48038350 msgr2=0x7f5d4803a810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/835547951 2024-01-24T13:30:10.179 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.562+0000 7f5d5e7fc700 1 --2- 172.21.15.29:0/179038770 >> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] conn(0x7f5d48038350 0x7f5d4803a810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-24T13:30:10.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.763+0000 7f5d5e7fc700 1 -- 172.21.15.29:0/179038770 >> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] conn(0x7f5d48038350 msgr2=0x7f5d4803a810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/835547951 2024-01-24T13:30:10.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:04.763+0000 7f5d5e7fc700 1 --2- 172.21.15.29:0/179038770 >> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] conn(0x7f5d48038350 0x7f5d4803a810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-24T13:30:10.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:05.164+0000 7f5d5e7fc700 1 -- 172.21.15.29:0/179038770 >> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] conn(0x7f5d48038350 msgr2=0x7f5d4803a810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/835547951 2024-01-24T13:30:10.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:05.164+0000 7f5d5e7fc700 1 --2- 172.21.15.29:0/179038770 >> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] conn(0x7f5d48038350 0x7f5d4803a810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-24T13:30:10.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:05.965+0000 7f5d5e7fc700 1 -- 172.21.15.29:0/179038770 >> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] conn(0x7f5d48038350 msgr2=0x7f5d4803a810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/835547951 2024-01-24T13:30:10.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:05.965+0000 7f5d5e7fc700 1 --2- 172.21.15.29:0/179038770 >> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] conn(0x7f5d48038350 0x7f5d4803a810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-24T13:30:10.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:07.566+0000 7f5d5e7fc700 1 -- 172.21.15.29:0/179038770 >> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] conn(0x7f5d48038350 msgr2=0x7f5d4803a810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/835547951 2024-01-24T13:30:10.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:07.566+0000 7f5d5e7fc700 1 --2- 172.21.15.29:0/179038770 >> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] conn(0x7f5d48038350 0x7f5d4803a810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-01-24T13:30:10.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:08.776+0000 7f5d47fff700 1 -- 172.21.15.29:0/179038770 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mgrmap(e 6) v1 ==== 45035+0+0 (secure 0 0 0) 0x7f5d5002d910 con 0x7f5d60104790 2024-01-24T13:30:10.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:08.776+0000 7f5d47fff700 1 -- 172.21.15.29:0/179038770 >> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] conn(0x7f5d48038350 msgr2=0x7f5d4803a810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-24T13:30:10.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:08.776+0000 7f5d47fff700 1 --2- 172.21.15.29:0/179038770 >> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] conn(0x7f5d48038350 0x7f5d4803a810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:10.180 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:09.778+0000 7f5d47fff700 1 -- 172.21.15.29:0/179038770 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 7) v1 ==== 45162+0+0 (secure 0 0 0) 0x7f5d5002ecf0 con 0x7f5d60104790 2024-01-24T13:30:10.181 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:09.778+0000 7f5d47fff700 1 --2- 172.21.15.29:0/179038770 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f5d4803bc30 0x7f5d4803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:10.181 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:09.778+0000 7f5d47fff700 1 -- 172.21.15.29:0/179038770 --> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5d4803af20 con 0x7f5d4803bc30 2024-01-24T13:30:10.181 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:09.779+0000 7f5d5e7fc700 1 --2- 172.21.15.29:0/179038770 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f5d4803bc30 0x7f5d4803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:10.181 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:09.780+0000 7f5d5e7fc700 1 --2- 172.21.15.29:0/179038770 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f5d4803bc30 0x7f5d4803e020 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f5d54003a10 tx=0x7f5d540092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:10.181 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:09.781+0000 7f5d47fff700 1 -- 172.21.15.29:0/179038770 <== mgr.14120 v2:172.21.15.29:6800/1996568496 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f5d4803af20 con 0x7f5d4803bc30 2024-01-24T13:30:10.181 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:09.794+0000 7f5d65621700 1 -- 172.21.15.29:0/179038770 --> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f5d4c002630 con 0x7f5d4803bc30 2024-01-24T13:30:10.181 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:09.794+0000 7f5d47fff700 1 -- 172.21.15.29:0/179038770 <== mgr.14120 v2:172.21.15.29:6800/1996568496 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f5d4c002630 con 0x7f5d4803bc30 2024-01-24T13:30:10.181 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:09.794+0000 7f5d65621700 1 -- 172.21.15.29:0/179038770 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f5d4803bc30 msgr2=0x7f5d4803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:10.181 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:09.794+0000 7f5d65621700 1 --2- 172.21.15.29:0/179038770 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f5d4803bc30 0x7f5d4803e020 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f5d54003a10 tx=0x7f5d540092b0 comp rx=0 tx=0).stop 2024-01-24T13:30:10.181 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:09.794+0000 7f5d65621700 1 -- 172.21.15.29:0/179038770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d60104790 msgr2=0x7f5d6019a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:10.181 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:09.794+0000 7f5d65621700 1 --2- 172.21.15.29:0/179038770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d60104790 0x7f5d6019a100 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5d50006af0 tx=0x7f5d50011ad0 comp rx=0 tx=0).stop 2024-01-24T13:30:10.181 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:09.794+0000 7f5d65621700 1 -- 172.21.15.29:0/179038770 shutdown_connections 2024-01-24T13:30:10.182 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:09.794+0000 7f5d65621700 1 --2- 172.21.15.29:0/179038770 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f5d4803bc30 0x7f5d4803e020 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:10.182 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:09.794+0000 7f5d65621700 1 --2- 172.21.15.29:0/179038770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d60104790 0x7f5d6019a100 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:10.182 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:09.794+0000 7f5d65621700 1 --2- 172.21.15.29:0/179038770 >> [v2:172.21.15.29:6800/835547951,v1:172.21.15.29:6801/835547951] conn(0x7f5d48038350 0x7f5d4803a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:10.182 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:09.794+0000 7f5d65621700 1 -- 172.21.15.29:0/179038770 >> 172.21.15.29:0/179038770 conn(0x7f5d60100140 msgr2=0x7f5d60100eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:10.182 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:09.795+0000 7f5d65621700 1 -- 172.21.15.29:0/179038770 shutdown_connections 2024-01-24T13:30:10.182 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:09.795+0000 7f5d65621700 1 -- 172.21.15.29:0/179038770 wait complete. 2024-01-24T13:30:10.182 INFO:teuthology.orchestra.run.smithi029.stdout:mgr epoch 5 is available 2024-01-24T13:30:10.182 INFO:teuthology.orchestra.run.smithi029.stdout:Setting orchestrator backend to cephadm... 2024-01-24T13:30:11.085 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:11 smithi029 ceph-mon[40651]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-24T13:30:11.085 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:11 smithi029 ceph-mon[40651]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-24T13:30:11.085 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:11 smithi029 ceph-mon[40651]: from='mgr.14120 172.21.15.29:0/656713355' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:30:11.085 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:11 smithi029 ceph-mon[40651]: from='mgr.14120 172.21.15.29:0/656713355' entity='mgr.x' 2024-01-24T13:30:11.086 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:11 smithi029 ceph-mon[40651]: from='mgr.14120 172.21.15.29:0/656713355' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:30:11.117 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.531+0000 7fb5e2801700 1 Processor -- start 2024-01-24T13:30:11.117 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.531+0000 7fb5e2801700 1 -- start start 2024-01-24T13:30:11.118 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.532+0000 7fb5e2801700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5dc106f30 0x7fb5dc109320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:11.118 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.532+0000 7fb5e2801700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5dc079e50 con 0x7fb5dc106f30 2024-01-24T13:30:11.118 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.532+0000 7fb5dbfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5dc106f30 0x7fb5dc109320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:11.118 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.532+0000 7fb5dbfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5dc106f30 0x7fb5dc109320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46870/0 (socket says 172.21.15.29:46870) 2024-01-24T13:30:11.118 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.532+0000 7fb5dbfff700 1 -- 172.21.15.29:0/1385610606 learned_addr learned my addr 172.21.15.29:0/1385610606 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:11.118 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.532+0000 7fb5dbfff700 1 -- 172.21.15.29:0/1385610606 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5dc109860 con 0x7fb5dc106f30 2024-01-24T13:30:11.118 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.532+0000 7fb5dbfff700 1 --2- 172.21.15.29:0/1385610606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5dc106f30 0x7fb5dc109320 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc0098a0 tx=0x7fb5cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=571a9578c82d7fec server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:11.118 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.533+0000 7fb5daffd700 1 -- 172.21.15.29:0/1385610606 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb5cc004160 con 0x7fb5dc106f30 2024-01-24T13:30:11.118 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.533+0000 7fb5daffd700 1 -- 172.21.15.29:0/1385610606 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb5cc0042c0 con 0x7fb5dc106f30 2024-01-24T13:30:11.118 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.533+0000 7fb5daffd700 1 -- 172.21.15.29:0/1385610606 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb5cc011a00 con 0x7fb5dc106f30 2024-01-24T13:30:11.118 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.533+0000 7fb5e2801700 1 -- 172.21.15.29:0/1385610606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5dc106f30 msgr2=0x7fb5dc109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:11.118 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.533+0000 7fb5e2801700 1 --2- 172.21.15.29:0/1385610606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5dc106f30 0x7fb5dc109320 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc0098a0 tx=0x7fb5cc009bb0 comp rx=0 tx=0).stop 2024-01-24T13:30:11.119 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.533+0000 7fb5e2801700 1 -- 172.21.15.29:0/1385610606 shutdown_connections 2024-01-24T13:30:11.119 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.533+0000 7fb5e2801700 1 --2- 172.21.15.29:0/1385610606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5dc106f30 0x7fb5dc109320 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:11.119 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.533+0000 7fb5e2801700 1 -- 172.21.15.29:0/1385610606 >> 172.21.15.29:0/1385610606 conn(0x7fb5dc100940 msgr2=0x7fb5dc102d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:11.119 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.533+0000 7fb5e2801700 1 -- 172.21.15.29:0/1385610606 shutdown_connections 2024-01-24T13:30:11.119 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.533+0000 7fb5e2801700 1 -- 172.21.15.29:0/1385610606 wait complete. 2024-01-24T13:30:11.119 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.534+0000 7fb5e2801700 1 Processor -- start 2024-01-24T13:30:11.119 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.535+0000 7fb5e2801700 1 -- start start 2024-01-24T13:30:11.119 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.535+0000 7fb5e2801700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5dc106f30 0x7fb5dc19a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:11.119 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.535+0000 7fb5e2801700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5dc19a670 con 0x7fb5dc106f30 2024-01-24T13:30:11.119 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.535+0000 7fb5dbfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5dc106f30 0x7fb5dc19a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:11.119 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.535+0000 7fb5dbfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5dc106f30 0x7fb5dc19a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46884/0 (socket says 172.21.15.29:46884) 2024-01-24T13:30:11.119 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.535+0000 7fb5dbfff700 1 -- 172.21.15.29:0/3876231035 learned_addr learned my addr 172.21.15.29:0/3876231035 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:11.119 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.535+0000 7fb5dbfff700 1 -- 172.21.15.29:0/3876231035 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5cc009550 con 0x7fb5dc106f30 2024-01-24T13:30:11.119 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.536+0000 7fb5dbfff700 1 --2- 172.21.15.29:0/3876231035 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5dc106f30 0x7fb5dc19a130 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc009fb0 tx=0x7fb5cc011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:11.120 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.536+0000 7fb5d97fa700 1 -- 172.21.15.29:0/3876231035 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb5cc003a40 con 0x7fb5dc106f30 2024-01-24T13:30:11.120 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.536+0000 7fb5d97fa700 1 -- 172.21.15.29:0/3876231035 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb5cc003ba0 con 0x7fb5dc106f30 2024-01-24T13:30:11.120 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.536+0000 7fb5e2801700 1 -- 172.21.15.29:0/3876231035 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5dc19a870 con 0x7fb5dc106f30 2024-01-24T13:30:11.120 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.536+0000 7fb5d97fa700 1 -- 172.21.15.29:0/3876231035 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb5cc01a750 con 0x7fb5dc106f30 2024-01-24T13:30:11.120 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.536+0000 7fb5e2801700 1 -- 172.21.15.29:0/3876231035 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5dc19ad10 con 0x7fb5dc106f30 2024-01-24T13:30:11.120 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.537+0000 7fb5d97fa700 1 -- 172.21.15.29:0/3876231035 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 7) v1 ==== 45162+0+0 (secure 0 0 0) 0x7fb5cc024430 con 0x7fb5dc106f30 2024-01-24T13:30:11.120 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.537+0000 7fb5e2801700 1 -- 172.21.15.29:0/3876231035 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5dc10a600 con 0x7fb5dc106f30 2024-01-24T13:30:11.120 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.537+0000 7fb5d97fa700 1 --2- 172.21.15.29:0/3876231035 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7fb5c4038190 0x7fb5c403a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:11.120 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.538+0000 7fb5d97fa700 1 -- 172.21.15.29:0/3876231035 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb5cc04c0b0 con 0x7fb5dc106f30 2024-01-24T13:30:11.120 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.538+0000 7fb5db7fe700 1 --2- 172.21.15.29:0/3876231035 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7fb5c4038190 0x7fb5c403a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:11.120 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.538+0000 7fb5db7fe700 1 --2- 172.21.15.29:0/3876231035 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7fb5c4038190 0x7fb5c403a650 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0006fd0 tx=0x7fb5d0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:11.120 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.543+0000 7fb5d97fa700 1 -- 172.21.15.29:0/3876231035 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fb5cc0189a0 con 0x7fb5dc106f30 2024-01-24T13:30:11.120 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.767+0000 7fb5e2801700 1 -- 172.21.15.29:0/3876231035 --> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fb5dc102bf0 con 0x7fb5c4038190 2024-01-24T13:30:11.120 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.772+0000 7fb5d97fa700 1 -- 172.21.15.29:0/3876231035 <== mgr.14120 v2:172.21.15.29:6800/1996568496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb5dc102bf0 con 0x7fb5c4038190 2024-01-24T13:30:11.121 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.775+0000 7fb5e2801700 1 -- 172.21.15.29:0/3876231035 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7fb5c4038190 msgr2=0x7fb5c403a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:11.121 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.775+0000 7fb5e2801700 1 --2- 172.21.15.29:0/3876231035 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7fb5c4038190 0x7fb5c403a650 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0006fd0 tx=0x7fb5d0006e40 comp rx=0 tx=0).stop 2024-01-24T13:30:11.121 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.775+0000 7fb5e2801700 1 -- 172.21.15.29:0/3876231035 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5dc106f30 msgr2=0x7fb5dc19a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:11.121 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.775+0000 7fb5e2801700 1 --2- 172.21.15.29:0/3876231035 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5dc106f30 0x7fb5dc19a130 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc009fb0 tx=0x7fb5cc011ad0 comp rx=0 tx=0).stop 2024-01-24T13:30:11.121 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.776+0000 7fb5e2801700 1 -- 172.21.15.29:0/3876231035 shutdown_connections 2024-01-24T13:30:11.121 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.776+0000 7fb5e2801700 1 --2- 172.21.15.29:0/3876231035 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7fb5c4038190 0x7fb5c403a650 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:11.121 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.776+0000 7fb5e2801700 1 --2- 172.21.15.29:0/3876231035 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5dc106f30 0x7fb5dc19a130 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:11.121 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.776+0000 7fb5e2801700 1 -- 172.21.15.29:0/3876231035 >> 172.21.15.29:0/3876231035 conn(0x7fb5dc100940 msgr2=0x7fb5dc101f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:11.121 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.776+0000 7fb5e2801700 1 -- 172.21.15.29:0/3876231035 shutdown_connections 2024-01-24T13:30:11.121 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:10.776+0000 7fb5e2801700 1 -- 172.21.15.29:0/3876231035 wait complete. 2024-01-24T13:30:12.074 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout value unchanged 2024-01-24T13:30:12.080 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.492+0000 7f86cc5c1700 1 Processor -- start 2024-01-24T13:30:12.080 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.492+0000 7f86cc5c1700 1 -- start start 2024-01-24T13:30:12.080 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.493+0000 7f86cc5c1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86c4105360 0x7f86c4105780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:12.080 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.493+0000 7f86cc5c1700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c4105d50 con 0x7f86c4105360 2024-01-24T13:30:12.080 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.493+0000 7f86ca35d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86c4105360 0x7f86c4105780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:12.080 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.493+0000 7f86ca35d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86c4105360 0x7f86c4105780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43824/0 (socket says 172.21.15.29:43824) 2024-01-24T13:30:12.080 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.493+0000 7f86ca35d700 1 -- 172.21.15.29:0/135753182 learned_addr learned my addr 172.21.15.29:0/135753182 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:12.080 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.494+0000 7f86ca35d700 1 -- 172.21.15.29:0/135753182 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c4106560 con 0x7f86c4105360 2024-01-24T13:30:12.081 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.494+0000 7f86ca35d700 1 --2- 172.21.15.29:0/135753182 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86c4105360 0x7f86c4105780 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f86c0009e80 tx=0x7f86c000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=671b418bc06d06d4 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:12.081 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.494+0000 7f86c935b700 1 -- 172.21.15.29:0/135753182 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f86c000e070 con 0x7f86c4105360 2024-01-24T13:30:12.081 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.494+0000 7f86c935b700 1 -- 172.21.15.29:0/135753182 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f86c000b8c0 con 0x7f86c4105360 2024-01-24T13:30:12.081 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.494+0000 7f86c935b700 1 -- 172.21.15.29:0/135753182 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f86c00044c0 con 0x7f86c4105360 2024-01-24T13:30:12.081 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.495+0000 7f86cc5c1700 1 -- 172.21.15.29:0/135753182 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86c4105360 msgr2=0x7f86c4105780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:12.081 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.495+0000 7f86cc5c1700 1 --2- 172.21.15.29:0/135753182 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86c4105360 0x7f86c4105780 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f86c0009e80 tx=0x7f86c000b330 comp rx=0 tx=0).stop 2024-01-24T13:30:12.081 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.495+0000 7f86cc5c1700 1 -- 172.21.15.29:0/135753182 shutdown_connections 2024-01-24T13:30:12.081 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.495+0000 7f86cc5c1700 1 --2- 172.21.15.29:0/135753182 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86c4105360 0x7f86c4105780 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:12.081 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.495+0000 7f86cc5c1700 1 -- 172.21.15.29:0/135753182 >> 172.21.15.29:0/135753182 conn(0x7f86c4100980 msgr2=0x7f86c4102da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:12.081 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.495+0000 7f86cc5c1700 1 -- 172.21.15.29:0/135753182 shutdown_connections 2024-01-24T13:30:12.081 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.495+0000 7f86cc5c1700 1 -- 172.21.15.29:0/135753182 wait complete. 2024-01-24T13:30:12.081 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.495+0000 7f86cc5c1700 1 Processor -- start 2024-01-24T13:30:12.081 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.495+0000 7f86cc5c1700 1 -- start start 2024-01-24T13:30:12.081 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.496+0000 7f86cc5c1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86c4105360 0x7f86c419c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:12.082 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.496+0000 7f86cc5c1700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c419c7f0 con 0x7f86c4105360 2024-01-24T13:30:12.082 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.496+0000 7f86ca35d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86c4105360 0x7f86c419c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:12.082 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.496+0000 7f86ca35d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86c4105360 0x7f86c419c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43840/0 (socket says 172.21.15.29:43840) 2024-01-24T13:30:12.082 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.496+0000 7f86ca35d700 1 -- 172.21.15.29:0/2819432019 learned_addr learned my addr 172.21.15.29:0/2819432019 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:12.082 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.496+0000 7f86ca35d700 1 -- 172.21.15.29:0/2819432019 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c0009b30 con 0x7f86c4105360 2024-01-24T13:30:12.082 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.496+0000 7f86ca35d700 1 --2- 172.21.15.29:0/2819432019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86c4105360 0x7f86c419c2b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f86c0009e50 tx=0x7f86c0004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:12.082 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.496+0000 7f86bb7fe700 1 -- 172.21.15.29:0/2819432019 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f86c000e040 con 0x7f86c4105360 2024-01-24T13:30:12.082 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.496+0000 7f86cc5c1700 1 -- 172.21.15.29:0/2819432019 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86c419c9f0 con 0x7f86c4105360 2024-01-24T13:30:12.082 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.496+0000 7f86bb7fe700 1 -- 172.21.15.29:0/2819432019 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f86c000bd70 con 0x7f86c4105360 2024-01-24T13:30:12.082 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.496+0000 7f86bb7fe700 1 -- 172.21.15.29:0/2819432019 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f86c001c780 con 0x7f86c4105360 2024-01-24T13:30:12.082 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.497+0000 7f86cc5c1700 1 -- 172.21.15.29:0/2819432019 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86c419cef0 con 0x7f86c4105360 2024-01-24T13:30:12.082 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.498+0000 7f86cc5c1700 1 -- 172.21.15.29:0/2819432019 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86c4062370 con 0x7f86c4105360 2024-01-24T13:30:12.082 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.498+0000 7f86bb7fe700 1 -- 172.21.15.29:0/2819432019 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 7) v1 ==== 45162+0+0 (secure 0 0 0) 0x7f86c0012430 con 0x7f86c4105360 2024-01-24T13:30:12.082 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.498+0000 7f86bb7fe700 1 --2- 172.21.15.29:0/2819432019 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f86b0038190 0x7f86b003a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:12.082 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.499+0000 7f86bb7fe700 1 -- 172.21.15.29:0/2819432019 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f86c004c0d0 con 0x7f86c4105360 2024-01-24T13:30:12.083 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.499+0000 7f86c9b5c700 1 --2- 172.21.15.29:0/2819432019 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f86b0038190 0x7f86b003a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:12.083 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.499+0000 7f86c9b5c700 1 --2- 172.21.15.29:0/2819432019 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f86b0038190 0x7f86b003a650 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f86b4006fd0 tx=0x7f86b4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:12.083 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.503+0000 7f86bb7fe700 1 -- 172.21.15.29:0/2819432019 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f86c0012ad0 con 0x7f86c4105360 2024-01-24T13:30:12.083 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.710+0000 7f86cc5c1700 1 -- 172.21.15.29:0/2819432019 --> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f86c4102ad0 con 0x7f86b0038190 2024-01-24T13:30:12.083 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.710+0000 7f86bb7fe700 1 -- 172.21.15.29:0/2819432019 <== mgr.14120 v2:172.21.15.29:6800/1996568496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f86c4102ad0 con 0x7f86b0038190 2024-01-24T13:30:12.083 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.712+0000 7f86cc5c1700 1 -- 172.21.15.29:0/2819432019 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f86b0038190 msgr2=0x7f86b003a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:12.083 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.712+0000 7f86cc5c1700 1 --2- 172.21.15.29:0/2819432019 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f86b0038190 0x7f86b003a650 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f86b4006fd0 tx=0x7f86b4006e40 comp rx=0 tx=0).stop 2024-01-24T13:30:12.083 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.712+0000 7f86cc5c1700 1 -- 172.21.15.29:0/2819432019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86c4105360 msgr2=0x7f86c419c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:12.083 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.712+0000 7f86cc5c1700 1 --2- 172.21.15.29:0/2819432019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86c4105360 0x7f86c419c2b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f86c0009e50 tx=0x7f86c0004610 comp rx=0 tx=0).stop 2024-01-24T13:30:12.083 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.713+0000 7f86cc5c1700 1 -- 172.21.15.29:0/2819432019 shutdown_connections 2024-01-24T13:30:12.083 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.713+0000 7f86cc5c1700 1 --2- 172.21.15.29:0/2819432019 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f86b0038190 0x7f86b003a650 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:12.083 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.713+0000 7f86cc5c1700 1 --2- 172.21.15.29:0/2819432019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86c4105360 0x7f86c419c2b0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:12.083 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.713+0000 7f86cc5c1700 1 -- 172.21.15.29:0/2819432019 >> 172.21.15.29:0/2819432019 conn(0x7f86c4100980 msgr2=0x7f86c4079940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:12.083 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.713+0000 7f86cc5c1700 1 -- 172.21.15.29:0/2819432019 shutdown_connections 2024-01-24T13:30:12.084 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:11.713+0000 7f86cc5c1700 1 -- 172.21.15.29:0/2819432019 wait complete. 2024-01-24T13:30:12.084 INFO:teuthology.orchestra.run.smithi029.stdout:Generating ssh key... 2024-01-24T13:30:12.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:12 smithi029 ceph-mon[40651]: [24/Jan/2024:13:30:10] ENGINE Bus STARTING 2024-01-24T13:30:12.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:12 smithi029 ceph-mon[40651]: [24/Jan/2024:13:30:10] ENGINE Serving on https://172.21.15.29:7150 2024-01-24T13:30:12.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:12 smithi029 ceph-mon[40651]: [24/Jan/2024:13:30:10] ENGINE Serving on http://172.21.15.29:8765 2024-01-24T13:30:12.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:12 smithi029 ceph-mon[40651]: [24/Jan/2024:13:30:10] ENGINE Bus STARTED 2024-01-24T13:30:12.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:12 smithi029 ceph-mon[40651]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:30:12.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:12 smithi029 ceph-mon[40651]: mgrmap e8: x(active, since 2s) 2024-01-24T13:30:13.334 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:13 smithi029 ceph-mon[40651]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:30:13.335 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:13 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: Generating public/private rsa key pair. 2024-01-24T13:30:13.335 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:13 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: Your identification has been saved in /tmp/tmp4lq41xz6/key. 2024-01-24T13:30:13.335 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:13 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: Your public key has been saved in /tmp/tmp4lq41xz6/key.pub. 2024-01-24T13:30:13.335 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:13 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: The key fingerprint is: 2024-01-24T13:30:13.335 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:13 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: SHA256:W6a1l7wr1yd5R3kA2HlYFH6FHUsWAhy4Wa5FK+NXYD8 ceph-83afcf5e-babc-11ee-95b1-87774f69a715 2024-01-24T13:30:13.335 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:13 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: The key's randomart image is: 2024-01-24T13:30:13.335 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:13 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: +---[RSA 3072]----+ 2024-01-24T13:30:13.335 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:13 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: | o=o*+O+| 2024-01-24T13:30:13.335 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:13 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: | ..*=.* +| 2024-01-24T13:30:13.335 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:13 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: | B +o...| 2024-01-24T13:30:13.336 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:13 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: | = + E.. | 2024-01-24T13:30:13.336 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:13 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: | S O . ...| 2024-01-24T13:30:13.336 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:13 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: | O + . .o| 2024-01-24T13:30:13.336 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:13 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: | o o +. o.| 2024-01-24T13:30:13.336 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:13 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: | ....+ +| 2024-01-24T13:30:13.336 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:13 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: | oo. +.| 2024-01-24T13:30:13.336 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:13 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: +----[SHA256]-----+ 2024-01-24T13:30:13.407 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.482+0000 7ff4c17dd700 1 Processor -- start 2024-01-24T13:30:13.407 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.482+0000 7ff4c17dd700 1 -- start start 2024-01-24T13:30:13.407 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.482+0000 7ff4c17dd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4bc1058c0 0x7ff4bc105ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:13.407 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.482+0000 7ff4c17dd700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4bc1062b0 con 0x7ff4bc1058c0 2024-01-24T13:30:13.408 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.482+0000 7ff4baffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4bc1058c0 0x7ff4bc105ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:13.408 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.482+0000 7ff4baffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4bc1058c0 0x7ff4bc105ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43854/0 (socket says 172.21.15.29:43854) 2024-01-24T13:30:13.408 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.482+0000 7ff4baffd700 1 -- 172.21.15.29:0/3303432509 learned_addr learned my addr 172.21.15.29:0/3303432509 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:13.408 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.483+0000 7ff4baffd700 1 -- 172.21.15.29:0/3303432509 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4bc1065e0 con 0x7ff4bc1058c0 2024-01-24T13:30:13.408 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.483+0000 7ff4baffd700 1 --2- 172.21.15.29:0/3303432509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4bc1058c0 0x7ff4bc105ce0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac0098a0 tx=0x7ff4ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc63ef47d858755e server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:13.408 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.483+0000 7ff4b9ffb700 1 -- 172.21.15.29:0/3303432509 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff4ac004160 con 0x7ff4bc1058c0 2024-01-24T13:30:13.408 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.483+0000 7ff4b9ffb700 1 -- 172.21.15.29:0/3303432509 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff4ac0042c0 con 0x7ff4bc1058c0 2024-01-24T13:30:13.408 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.483+0000 7ff4b9ffb700 1 -- 172.21.15.29:0/3303432509 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff4ac011a00 con 0x7ff4bc1058c0 2024-01-24T13:30:13.408 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.484+0000 7ff4c17dd700 1 -- 172.21.15.29:0/3303432509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4bc1058c0 msgr2=0x7ff4bc105ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:13.409 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.484+0000 7ff4c17dd700 1 --2- 172.21.15.29:0/3303432509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4bc1058c0 0x7ff4bc105ce0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac0098a0 tx=0x7ff4ac009bb0 comp rx=0 tx=0).stop 2024-01-24T13:30:13.409 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.484+0000 7ff4c17dd700 1 -- 172.21.15.29:0/3303432509 shutdown_connections 2024-01-24T13:30:13.409 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.484+0000 7ff4c17dd700 1 --2- 172.21.15.29:0/3303432509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4bc1058c0 0x7ff4bc105ce0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:13.409 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.484+0000 7ff4c17dd700 1 -- 172.21.15.29:0/3303432509 >> 172.21.15.29:0/3303432509 conn(0x7ff4bc100140 msgr2=0x7ff4bc102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:13.409 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.484+0000 7ff4c17dd700 1 -- 172.21.15.29:0/3303432509 shutdown_connections 2024-01-24T13:30:13.409 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.484+0000 7ff4c17dd700 1 -- 172.21.15.29:0/3303432509 wait complete. 2024-01-24T13:30:13.409 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.485+0000 7ff4c17dd700 1 Processor -- start 2024-01-24T13:30:13.409 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.485+0000 7ff4c17dd700 1 -- start start 2024-01-24T13:30:13.409 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.485+0000 7ff4c17dd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4bc1058c0 0x7ff4bc190540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:13.409 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.485+0000 7ff4c17dd700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4bc190a80 con 0x7ff4bc1058c0 2024-01-24T13:30:13.409 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.485+0000 7ff4baffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4bc1058c0 0x7ff4bc190540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:13.409 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.485+0000 7ff4baffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4bc1058c0 0x7ff4bc190540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43866/0 (socket says 172.21.15.29:43866) 2024-01-24T13:30:13.409 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.485+0000 7ff4baffd700 1 -- 172.21.15.29:0/4248607790 learned_addr learned my addr 172.21.15.29:0/4248607790 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:13.409 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.486+0000 7ff4baffd700 1 -- 172.21.15.29:0/4248607790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4ac009550 con 0x7ff4bc1058c0 2024-01-24T13:30:13.409 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.486+0000 7ff4baffd700 1 --2- 172.21.15.29:0/4248607790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4bc1058c0 0x7ff4bc190540 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac009fb0 tx=0x7ff4ac011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:13.410 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.486+0000 7ff4a3fff700 1 -- 172.21.15.29:0/4248607790 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff4ac003a40 con 0x7ff4bc1058c0 2024-01-24T13:30:13.410 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.486+0000 7ff4a3fff700 1 -- 172.21.15.29:0/4248607790 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff4ac003ba0 con 0x7ff4bc1058c0 2024-01-24T13:30:13.410 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.486+0000 7ff4c17dd700 1 -- 172.21.15.29:0/4248607790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4bc18eb90 con 0x7ff4bc1058c0 2024-01-24T13:30:13.410 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.486+0000 7ff4a3fff700 1 -- 172.21.15.29:0/4248607790 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff4ac01a750 con 0x7ff4bc1058c0 2024-01-24T13:30:13.410 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.486+0000 7ff4c17dd700 1 -- 172.21.15.29:0/4248607790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4bc18f030 con 0x7ff4bc1058c0 2024-01-24T13:30:13.410 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.487+0000 7ff4c17dd700 1 -- 172.21.15.29:0/4248607790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4bc062370 con 0x7ff4bc1058c0 2024-01-24T13:30:13.410 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.487+0000 7ff4a3fff700 1 -- 172.21.15.29:0/4248607790 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 45268+0+0 (secure 0 0 0) 0x7ff4ac024b60 con 0x7ff4bc1058c0 2024-01-24T13:30:13.410 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.488+0000 7ff4a3fff700 1 --2- 172.21.15.29:0/4248607790 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7ff4a4038300 0x7ff4a403a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:13.410 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.488+0000 7ff4a3fff700 1 -- 172.21.15.29:0/4248607790 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff4ac04c1a0 con 0x7ff4bc1058c0 2024-01-24T13:30:13.410 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.488+0000 7ff4ba7fc700 1 --2- 172.21.15.29:0/4248607790 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7ff4a4038300 0x7ff4a403a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:13.410 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.488+0000 7ff4ba7fc700 1 --2- 172.21.15.29:0/4248607790 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7ff4a4038300 0x7ff4a403a7c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff4b0006fd0 tx=0x7ff4b0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:13.410 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.493+0000 7ff4a3fff700 1 -- 172.21.15.29:0/4248607790 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7ff4ac024430 con 0x7ff4bc1058c0 2024-01-24T13:30:13.410 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:12.701+0000 7ff4c17dd700 1 -- 172.21.15.29:0/4248607790 --> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4bc1022c0 con 0x7ff4a4038300 2024-01-24T13:30:13.410 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.047+0000 7ff4a3fff700 1 -- 172.21.15.29:0/4248607790 <== mgr.14120 v2:172.21.15.29:6800/1996568496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff4bc1022c0 con 0x7ff4a4038300 2024-01-24T13:30:13.411 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.049+0000 7ff4c17dd700 1 -- 172.21.15.29:0/4248607790 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7ff4a4038300 msgr2=0x7ff4a403a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:13.411 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.049+0000 7ff4c17dd700 1 --2- 172.21.15.29:0/4248607790 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7ff4a4038300 0x7ff4a403a7c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff4b0006fd0 tx=0x7ff4b0006e40 comp rx=0 tx=0).stop 2024-01-24T13:30:13.411 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.049+0000 7ff4c17dd700 1 -- 172.21.15.29:0/4248607790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4bc1058c0 msgr2=0x7ff4bc190540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:13.411 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.049+0000 7ff4c17dd700 1 --2- 172.21.15.29:0/4248607790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4bc1058c0 0x7ff4bc190540 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac009fb0 tx=0x7ff4ac011ad0 comp rx=0 tx=0).stop 2024-01-24T13:30:13.411 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.049+0000 7ff4c17dd700 1 -- 172.21.15.29:0/4248607790 shutdown_connections 2024-01-24T13:30:13.411 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.049+0000 7ff4c17dd700 1 --2- 172.21.15.29:0/4248607790 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7ff4a4038300 0x7ff4a403a7c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:13.411 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.049+0000 7ff4c17dd700 1 --2- 172.21.15.29:0/4248607790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4bc1058c0 0x7ff4bc190540 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:13.411 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.049+0000 7ff4c17dd700 1 -- 172.21.15.29:0/4248607790 >> 172.21.15.29:0/4248607790 conn(0x7ff4bc100140 msgr2=0x7ff4bc104d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:13.411 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.049+0000 7ff4c17dd700 1 -- 172.21.15.29:0/4248607790 shutdown_connections 2024-01-24T13:30:13.411 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.049+0000 7ff4c17dd700 1 -- 172.21.15.29:0/4248607790 wait complete. 2024-01-24T13:30:14.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:14 smithi029 ceph-mon[40651]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:30:14.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:14 smithi029 ceph-mon[40651]: Generating ssh key... 2024-01-24T13:30:14.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:14 smithi029 ceph-mon[40651]: from='mgr.14120 172.21.15.29:0/656713355' entity='mgr.x' 2024-01-24T13:30:14.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:14 smithi029 ceph-mon[40651]: from='mgr.14120 172.21.15.29:0/656713355' entity='mgr.x' 2024-01-24T13:30:14.439 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDqf55o8JDj6jP24erpNFQ8fci7GwghB0vocUucS9zXqshz89guKmvquPG2e0mtkDHjc3tYKITRAJDxxw6Y9oXwrOHx1qncYX70Et3dy/L7mLW8SkeHUXShfOEc4IN9okr6YNcMOsWyP32VdyyhZUsUNSUcQi6OqzXGWkFfCAOBM8sffqFhxmQ4NnknjMClbjWKmzb7QzNPVu3qoySGAKa/DgOLvtfGfbogJfKqp8yTn98FqnJNShUck64pJ7wHb9FlPy/jUV62GUziCDddc5/h/63G2OjznUyoA8x967lz76PQOf/tDWuoLyK6QpJsccsRwyRljzJGaUm8Ge0TmnOykkqRBKwTbAtobjWTlEz+gQS1CIE6jBldfyXHU5+mFXDs2h5LohsorNiMv/lNRKNhoPT2ulJRs6axKHQ7IvJcpk/seAhNwkej5Iqf6i88IUQgCGuDt7jhJGqA5Gk+Vm5qi7KUnyZNhxKSr6LHUPYpPV8g3PNyox+e97BbclQAvrE= ceph-83afcf5e-babc-11ee-95b1-87774f69a715 2024-01-24T13:30:14.439 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.844+0000 7f4cd7fff700 1 Processor -- start 2024-01-24T13:30:14.439 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.844+0000 7f4cd7fff700 1 -- start start 2024-01-24T13:30:14.439 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.844+0000 7f4cd7fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cd8104bc0 0x7f4cd8104fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:14.440 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.844+0000 7f4cd7fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd8079df0 con 0x7f4cd8104bc0 2024-01-24T13:30:14.440 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.844+0000 7f4cd6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cd8104bc0 0x7f4cd8104fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:14.440 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.844+0000 7f4cd6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cd8104bc0 0x7f4cd8104fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43878/0 (socket says 172.21.15.29:43878) 2024-01-24T13:30:14.440 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.844+0000 7f4cd6ffd700 1 -- 172.21.15.29:0/160479628 learned_addr learned my addr 172.21.15.29:0/160479628 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:14.440 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.845+0000 7f4cd6ffd700 1 -- 172.21.15.29:0/160479628 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd8079f30 con 0x7f4cd8104bc0 2024-01-24T13:30:14.440 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.845+0000 7f4cd6ffd700 1 --2- 172.21.15.29:0/160479628 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cd8104bc0 0x7f4cd8104fe0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4cc80098a0 tx=0x7f4cc8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a77a381af40cc1c9 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:14.440 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.845+0000 7f4cd5ffb700 1 -- 172.21.15.29:0/160479628 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4cc8004160 con 0x7f4cd8104bc0 2024-01-24T13:30:14.440 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.845+0000 7f4cd5ffb700 1 -- 172.21.15.29:0/160479628 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4cc80042c0 con 0x7f4cd8104bc0 2024-01-24T13:30:14.440 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.845+0000 7f4cd5ffb700 1 -- 172.21.15.29:0/160479628 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4cc8011a00 con 0x7f4cd8104bc0 2024-01-24T13:30:14.440 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.846+0000 7f4cd7fff700 1 -- 172.21.15.29:0/160479628 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cd8104bc0 msgr2=0x7f4cd8104fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:14.440 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.846+0000 7f4cd7fff700 1 --2- 172.21.15.29:0/160479628 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cd8104bc0 0x7f4cd8104fe0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4cc80098a0 tx=0x7f4cc8009bb0 comp rx=0 tx=0).stop 2024-01-24T13:30:14.440 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.846+0000 7f4cd7fff700 1 -- 172.21.15.29:0/160479628 shutdown_connections 2024-01-24T13:30:14.440 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.846+0000 7f4cd7fff700 1 --2- 172.21.15.29:0/160479628 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cd8104bc0 0x7f4cd8104fe0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:14.440 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.846+0000 7f4cd7fff700 1 -- 172.21.15.29:0/160479628 >> 172.21.15.29:0/160479628 conn(0x7f4cd81007a0 msgr2=0x7f4cd8102bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:14.441 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.846+0000 7f4cd7fff700 1 -- 172.21.15.29:0/160479628 shutdown_connections 2024-01-24T13:30:14.441 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.846+0000 7f4cd7fff700 1 -- 172.21.15.29:0/160479628 wait complete. 2024-01-24T13:30:14.441 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.847+0000 7f4cd7fff700 1 Processor -- start 2024-01-24T13:30:14.441 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.847+0000 7f4cd7fff700 1 -- start start 2024-01-24T13:30:14.441 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.847+0000 7f4cd7fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cd8104bc0 0x7f4cd819c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:14.441 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.847+0000 7f4cd7fff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd819c610 con 0x7f4cd8104bc0 2024-01-24T13:30:14.443 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.847+0000 7f4cd6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cd8104bc0 0x7f4cd819c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:14.443 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.847+0000 7f4cd6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cd8104bc0 0x7f4cd819c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43894/0 (socket says 172.21.15.29:43894) 2024-01-24T13:30:14.443 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.847+0000 7f4cd6ffd700 1 -- 172.21.15.29:0/1094075126 learned_addr learned my addr 172.21.15.29:0/1094075126 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:14.443 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.848+0000 7f4cd6ffd700 1 -- 172.21.15.29:0/1094075126 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc8009550 con 0x7f4cd8104bc0 2024-01-24T13:30:14.443 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.848+0000 7f4cd6ffd700 1 --2- 172.21.15.29:0/1094075126 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cd8104bc0 0x7f4cd819c0d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4cc8006af0 tx=0x7f4cc8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:14.443 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.848+0000 7f4cbffff700 1 -- 172.21.15.29:0/1094075126 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4cc8003a40 con 0x7f4cd8104bc0 2024-01-24T13:30:14.443 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.848+0000 7f4cbffff700 1 -- 172.21.15.29:0/1094075126 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4cc8003ba0 con 0x7f4cd8104bc0 2024-01-24T13:30:14.443 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.848+0000 7f4cbffff700 1 -- 172.21.15.29:0/1094075126 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4cc801a6e0 con 0x7f4cd8104bc0 2024-01-24T13:30:14.443 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.848+0000 7f4cd7fff700 1 -- 172.21.15.29:0/1094075126 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cd819c810 con 0x7f4cd8104bc0 2024-01-24T13:30:14.443 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.848+0000 7f4cd7fff700 1 -- 172.21.15.29:0/1094075126 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cd819ccb0 con 0x7f4cd8104bc0 2024-01-24T13:30:14.443 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.849+0000 7f4cbffff700 1 -- 172.21.15.29:0/1094075126 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 45268+0+0 (secure 0 0 0) 0x7f4cc801a860 con 0x7f4cd8104bc0 2024-01-24T13:30:14.443 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.849+0000 7f4cd7fff700 1 -- 172.21.15.29:0/1094075126 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cd8195950 con 0x7f4cd8104bc0 2024-01-24T13:30:14.443 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.849+0000 7f4cbffff700 1 --2- 172.21.15.29:0/1094075126 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f4cc0038350 0x7f4cc003a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:14.444 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.849+0000 7f4cbffff700 1 -- 172.21.15.29:0/1094075126 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f4cc804c1b0 con 0x7f4cd8104bc0 2024-01-24T13:30:14.444 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.849+0000 7f4cd67fc700 1 --2- 172.21.15.29:0/1094075126 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f4cc0038350 0x7f4cc003a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:14.444 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.850+0000 7f4cd67fc700 1 --2- 172.21.15.29:0/1094075126 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f4cc0038350 0x7f4cc003a810 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc006fd0 tx=0x7f4ccc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:14.444 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:13.854+0000 7f4cbffff700 1 -- 172.21.15.29:0/1094075126 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f4cc801ab10 con 0x7f4cd8104bc0 2024-01-24T13:30:14.444 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.063+0000 7f4cd7fff700 1 -- 172.21.15.29:0/1094075126 --> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f4cd8061180 con 0x7f4cc0038350 2024-01-24T13:30:14.444 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.064+0000 7f4cbffff700 1 -- 172.21.15.29:0/1094075126 <== mgr.14120 v2:172.21.15.29:6800/1996568496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f4cd8061180 con 0x7f4cc0038350 2024-01-24T13:30:14.444 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.066+0000 7f4cd7fff700 1 -- 172.21.15.29:0/1094075126 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f4cc0038350 msgr2=0x7f4cc003a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:14.444 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.066+0000 7f4cd7fff700 1 --2- 172.21.15.29:0/1094075126 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f4cc0038350 0x7f4cc003a810 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc006fd0 tx=0x7f4ccc006e40 comp rx=0 tx=0).stop 2024-01-24T13:30:14.444 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.066+0000 7f4cd7fff700 1 -- 172.21.15.29:0/1094075126 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cd8104bc0 msgr2=0x7f4cd819c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:14.444 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.066+0000 7f4cd7fff700 1 --2- 172.21.15.29:0/1094075126 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cd8104bc0 0x7f4cd819c0d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4cc8006af0 tx=0x7f4cc8011ad0 comp rx=0 tx=0).stop 2024-01-24T13:30:14.444 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.066+0000 7f4cd7fff700 1 -- 172.21.15.29:0/1094075126 shutdown_connections 2024-01-24T13:30:14.444 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.066+0000 7f4cd7fff700 1 --2- 172.21.15.29:0/1094075126 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f4cc0038350 0x7f4cc003a810 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:14.444 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.066+0000 7f4cd7fff700 1 --2- 172.21.15.29:0/1094075126 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cd8104bc0 0x7f4cd819c0d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:14.444 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.066+0000 7f4cd7fff700 1 -- 172.21.15.29:0/1094075126 >> 172.21.15.29:0/1094075126 conn(0x7f4cd81007a0 msgr2=0x7f4cd8190780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:14.444 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.067+0000 7f4cd7fff700 1 -- 172.21.15.29:0/1094075126 shutdown_connections 2024-01-24T13:30:14.445 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.067+0000 7f4cd7fff700 1 -- 172.21.15.29:0/1094075126 wait complete. 2024-01-24T13:30:14.445 INFO:teuthology.orchestra.run.smithi029.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-01-24T13:30:14.445 INFO:teuthology.orchestra.run.smithi029.stdout:Adding key to root@localhost authorized_keys... 2024-01-24T13:30:14.445 INFO:teuthology.orchestra.run.smithi029.stdout:Adding host smithi029... 2024-01-24T13:30:15.336 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:15 smithi029 ceph-mon[40651]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:30:16.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:16 smithi029 ceph-mon[40651]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi029", "addr": "172.21.15.29", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:30:17.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:17 smithi029 ceph-mon[40651]: Deploying cephadm binary to smithi029 2024-01-24T13:30:17.812 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Added host 'smithi029' with addr '172.21.15.29' 2024-01-24T13:30:17.812 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.810+0000 7f43c62ad700 1 Processor -- start 2024-01-24T13:30:17.812 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.810+0000 7f43c62ad700 1 -- start start 2024-01-24T13:30:17.812 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.810+0000 7f43c62ad700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43c0106ef0 0x7f43c01092e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:17.812 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.810+0000 7f43c62ad700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43c0079e90 con 0x7f43c0106ef0 2024-01-24T13:30:17.812 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.810+0000 7f43bffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43c0106ef0 0x7f43c01092e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:17.812 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.810+0000 7f43bffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43c0106ef0 0x7f43c01092e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43910/0 (socket says 172.21.15.29:43910) 2024-01-24T13:30:17.812 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.810+0000 7f43bffff700 1 -- 172.21.15.29:0/2303042667 learned_addr learned my addr 172.21.15.29:0/2303042667 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:17.813 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.811+0000 7f43bffff700 1 -- 172.21.15.29:0/2303042667 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43c0109820 con 0x7f43c0106ef0 2024-01-24T13:30:17.813 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.811+0000 7f43bffff700 1 --2- 172.21.15.29:0/2303042667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43c0106ef0 0x7f43c01092e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f43b00098a0 tx=0x7f43b0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee4e5cfd7c40605b server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:17.813 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.811+0000 7f43beffd700 1 -- 172.21.15.29:0/2303042667 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f43b0004160 con 0x7f43c0106ef0 2024-01-24T13:30:17.813 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.811+0000 7f43beffd700 1 -- 172.21.15.29:0/2303042667 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f43b00042c0 con 0x7f43c0106ef0 2024-01-24T13:30:17.813 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.811+0000 7f43beffd700 1 -- 172.21.15.29:0/2303042667 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f43b0011a00 con 0x7f43c0106ef0 2024-01-24T13:30:17.813 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.812+0000 7f43c62ad700 1 -- 172.21.15.29:0/2303042667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43c0106ef0 msgr2=0x7f43c01092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:17.813 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.812+0000 7f43c62ad700 1 --2- 172.21.15.29:0/2303042667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43c0106ef0 0x7f43c01092e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f43b00098a0 tx=0x7f43b0009bb0 comp rx=0 tx=0).stop 2024-01-24T13:30:17.813 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.812+0000 7f43c62ad700 1 -- 172.21.15.29:0/2303042667 shutdown_connections 2024-01-24T13:30:17.813 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.812+0000 7f43c62ad700 1 --2- 172.21.15.29:0/2303042667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43c0106ef0 0x7f43c01092e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:17.813 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.812+0000 7f43c62ad700 1 -- 172.21.15.29:0/2303042667 >> 172.21.15.29:0/2303042667 conn(0x7f43c0100940 msgr2=0x7f43c0102d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:17.813 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.812+0000 7f43c62ad700 1 -- 172.21.15.29:0/2303042667 shutdown_connections 2024-01-24T13:30:17.813 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.812+0000 7f43c62ad700 1 -- 172.21.15.29:0/2303042667 wait complete. 2024-01-24T13:30:17.813 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.813+0000 7f43c62ad700 1 Processor -- start 2024-01-24T13:30:17.813 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.813+0000 7f43c62ad700 1 -- start start 2024-01-24T13:30:17.813 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.813+0000 7f43c62ad700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43c0106ef0 0x7f43c019c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:17.814 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.813+0000 7f43c62ad700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43c019c840 con 0x7f43c0106ef0 2024-01-24T13:30:17.814 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.813+0000 7f43bffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43c0106ef0 0x7f43c019c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:17.814 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.813+0000 7f43bffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43c0106ef0 0x7f43c019c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43926/0 (socket says 172.21.15.29:43926) 2024-01-24T13:30:17.814 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.813+0000 7f43bffff700 1 -- 172.21.15.29:0/2347980859 learned_addr learned my addr 172.21.15.29:0/2347980859 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:17.814 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.813+0000 7f43bffff700 1 -- 172.21.15.29:0/2347980859 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b0009550 con 0x7f43c0106ef0 2024-01-24T13:30:17.814 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.814+0000 7f43bffff700 1 --2- 172.21.15.29:0/2347980859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43c0106ef0 0x7f43c019c300 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f43b0009fb0 tx=0x7f43b0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:17.814 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.814+0000 7f43bd7fa700 1 -- 172.21.15.29:0/2347980859 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f43b0003a40 con 0x7f43c0106ef0 2024-01-24T13:30:17.814 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.814+0000 7f43bd7fa700 1 -- 172.21.15.29:0/2347980859 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f43b0003ba0 con 0x7f43c0106ef0 2024-01-24T13:30:17.814 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.814+0000 7f43c62ad700 1 -- 172.21.15.29:0/2347980859 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43c019ca40 con 0x7f43c0106ef0 2024-01-24T13:30:17.814 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.814+0000 7f43bd7fa700 1 -- 172.21.15.29:0/2347980859 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f43b001a750 con 0x7f43c0106ef0 2024-01-24T13:30:17.814 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.814+0000 7f43c62ad700 1 -- 172.21.15.29:0/2347980859 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43c019cee0 con 0x7f43c0106ef0 2024-01-24T13:30:17.814 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.815+0000 7f43bd7fa700 1 -- 172.21.15.29:0/2347980859 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 45268+0+0 (secure 0 0 0) 0x7f43b001a9d0 con 0x7f43c0106ef0 2024-01-24T13:30:17.814 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.815+0000 7f43bd7fa700 1 --2- 172.21.15.29:0/2347980859 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f43a80382b0 0x7f43a803a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:17.814 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.815+0000 7f43bd7fa700 1 -- 172.21.15.29:0/2347980859 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f43b0024840 con 0x7f43c0106ef0 2024-01-24T13:30:17.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.815+0000 7f43bf7fe700 1 --2- 172.21.15.29:0/2347980859 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f43a80382b0 0x7f43a803a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:17.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.815+0000 7f43c62ad700 1 -- 172.21.15.29:0/2347980859 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43c0062370 con 0x7f43c0106ef0 2024-01-24T13:30:17.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.815+0000 7f43bf7fe700 1 --2- 172.21.15.29:0/2347980859 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f43a80382b0 0x7f43a803a770 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f43b4006fd0 tx=0x7f43b4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:17.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:14.820+0000 7f43bd7fa700 1 -- 172.21.15.29:0/2347980859 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f43b0023390 con 0x7f43c0106ef0 2024-01-24T13:30:17.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:15.036+0000 7f43c62ad700 1 -- 172.21.15.29:0/2347980859 --> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi029", "addr": "172.21.15.29", "target": ["mon-mgr", ""]}) v1 -- 0x7f43c0101df0 con 0x7f43a80382b0 2024-01-24T13:30:17.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:17.496+0000 7f43bd7fa700 1 -- 172.21.15.29:0/2347980859 <== mgr.14120 v2:172.21.15.29:6800/1996568496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f43c0101df0 con 0x7f43a80382b0 2024-01-24T13:30:17.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:17.499+0000 7f43c62ad700 1 -- 172.21.15.29:0/2347980859 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f43a80382b0 msgr2=0x7f43a803a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:17.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:17.499+0000 7f43c62ad700 1 --2- 172.21.15.29:0/2347980859 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f43a80382b0 0x7f43a803a770 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f43b4006fd0 tx=0x7f43b4006e40 comp rx=0 tx=0).stop 2024-01-24T13:30:17.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:17.499+0000 7f43c62ad700 1 -- 172.21.15.29:0/2347980859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43c0106ef0 msgr2=0x7f43c019c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:17.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:17.499+0000 7f43c62ad700 1 --2- 172.21.15.29:0/2347980859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43c0106ef0 0x7f43c019c300 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f43b0009fb0 tx=0x7f43b0011ad0 comp rx=0 tx=0).stop 2024-01-24T13:30:17.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:17.499+0000 7f43c62ad700 1 -- 172.21.15.29:0/2347980859 shutdown_connections 2024-01-24T13:30:17.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:17.499+0000 7f43c62ad700 1 --2- 172.21.15.29:0/2347980859 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f43a80382b0 0x7f43a803a770 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:17.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:17.499+0000 7f43c62ad700 1 --2- 172.21.15.29:0/2347980859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43c0106ef0 0x7f43c019c300 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:17.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:17.499+0000 7f43c62ad700 1 -- 172.21.15.29:0/2347980859 >> 172.21.15.29:0/2347980859 conn(0x7f43c0100940 msgr2=0x7f43c01016e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:17.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:17.499+0000 7f43c62ad700 1 -- 172.21.15.29:0/2347980859 shutdown_connections 2024-01-24T13:30:17.816 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:17.499+0000 7f43c62ad700 1 -- 172.21.15.29:0/2347980859 wait complete. 2024-01-24T13:30:17.816 INFO:teuthology.orchestra.run.smithi029.stdout:Deploying unmanaged mon service... 2024-01-24T13:30:18.676 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:18 smithi029 ceph-mon[40651]: from='mgr.14120 172.21.15.29:0/656713355' entity='mgr.x' 2024-01-24T13:30:18.677 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:18 smithi029 ceph-mon[40651]: Added host smithi029 2024-01-24T13:30:18.677 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:18 smithi029 ceph-mon[40651]: from='mgr.14120 172.21.15.29:0/656713355' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:30:18.997 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-01-24T13:30:18.998 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.262+0000 7f7ecd6ad700 1 Processor -- start 2024-01-24T13:30:18.998 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.262+0000 7f7ecd6ad700 1 -- start start 2024-01-24T13:30:18.998 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.262+0000 7f7ecd6ad700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ec8105380 0x7f7ec81057a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:18.998 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.262+0000 7f7ecd6ad700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec8105d70 con 0x7f7ec8105380 2024-01-24T13:30:18.998 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.262+0000 7f7ec6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ec8105380 0x7f7ec81057a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:18.998 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.262+0000 7f7ec6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ec8105380 0x7f7ec81057a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43934/0 (socket says 172.21.15.29:43934) 2024-01-24T13:30:18.998 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.262+0000 7f7ec6ffd700 1 -- 172.21.15.29:0/2173934628 learned_addr learned my addr 172.21.15.29:0/2173934628 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:18.998 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.263+0000 7f7ec6ffd700 1 -- 172.21.15.29:0/2173934628 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec8106580 con 0x7f7ec8105380 2024-01-24T13:30:18.998 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.263+0000 7f7ec6ffd700 1 --2- 172.21.15.29:0/2173934628 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ec8105380 0x7f7ec81057a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7eb80098a0 tx=0x7f7eb8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6674a3414d6d752 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:18.998 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.263+0000 7f7ec5ffb700 1 -- 172.21.15.29:0/2173934628 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7eb8004160 con 0x7f7ec8105380 2024-01-24T13:30:18.998 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.263+0000 7f7ec5ffb700 1 -- 172.21.15.29:0/2173934628 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7eb80042c0 con 0x7f7ec8105380 2024-01-24T13:30:18.998 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.263+0000 7f7ec5ffb700 1 -- 172.21.15.29:0/2173934628 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7eb8011a00 con 0x7f7ec8105380 2024-01-24T13:30:18.998 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.264+0000 7f7ecd6ad700 1 -- 172.21.15.29:0/2173934628 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ec8105380 msgr2=0x7f7ec81057a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:18.999 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.264+0000 7f7ecd6ad700 1 --2- 172.21.15.29:0/2173934628 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ec8105380 0x7f7ec81057a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7eb80098a0 tx=0x7f7eb8009bb0 comp rx=0 tx=0).stop 2024-01-24T13:30:18.999 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.264+0000 7f7ecd6ad700 1 -- 172.21.15.29:0/2173934628 shutdown_connections 2024-01-24T13:30:18.999 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.264+0000 7f7ecd6ad700 1 --2- 172.21.15.29:0/2173934628 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ec8105380 0x7f7ec81057a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:18.999 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.264+0000 7f7ecd6ad700 1 -- 172.21.15.29:0/2173934628 >> 172.21.15.29:0/2173934628 conn(0x7f7ec8100940 msgr2=0x7f7ec8102d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:18.999 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.264+0000 7f7ecd6ad700 1 -- 172.21.15.29:0/2173934628 shutdown_connections 2024-01-24T13:30:18.999 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.264+0000 7f7ecd6ad700 1 -- 172.21.15.29:0/2173934628 wait complete. 2024-01-24T13:30:18.999 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.265+0000 7f7ecd6ad700 1 Processor -- start 2024-01-24T13:30:18.999 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.265+0000 7f7ecd6ad700 1 -- start start 2024-01-24T13:30:18.999 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.265+0000 7f7ecd6ad700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ec8105380 0x7f7ec819a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:18.999 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.265+0000 7f7ecd6ad700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec819a670 con 0x7f7ec8105380 2024-01-24T13:30:18.999 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.265+0000 7f7ec6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ec8105380 0x7f7ec819a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:18.999 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.265+0000 7f7ec6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ec8105380 0x7f7ec819a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43938/0 (socket says 172.21.15.29:43938) 2024-01-24T13:30:18.999 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.265+0000 7f7ec6ffd700 1 -- 172.21.15.29:0/3622375151 learned_addr learned my addr 172.21.15.29:0/3622375151 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:19.000 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.266+0000 7f7ec6ffd700 1 -- 172.21.15.29:0/3622375151 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb8009550 con 0x7f7ec8105380 2024-01-24T13:30:19.000 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.266+0000 7f7ec6ffd700 1 --2- 172.21.15.29:0/3622375151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ec8105380 0x7f7ec819a130 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7eb8009130 tx=0x7f7eb8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:19.000 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.266+0000 7f7eaffff700 1 -- 172.21.15.29:0/3622375151 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7eb8003ab0 con 0x7f7ec8105380 2024-01-24T13:30:19.000 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.266+0000 7f7eaffff700 1 -- 172.21.15.29:0/3622375151 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7eb8003c10 con 0x7f7ec8105380 2024-01-24T13:30:19.000 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.266+0000 7f7ecd6ad700 1 -- 172.21.15.29:0/3622375151 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ec819a870 con 0x7f7ec8105380 2024-01-24T13:30:19.000 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.266+0000 7f7eaffff700 1 -- 172.21.15.29:0/3622375151 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7eb80247d0 con 0x7f7ec8105380 2024-01-24T13:30:19.000 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.266+0000 7f7ecd6ad700 1 -- 172.21.15.29:0/3622375151 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ec819ad10 con 0x7f7ec8105380 2024-01-24T13:30:19.000 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.267+0000 7f7eaffff700 1 -- 172.21.15.29:0/3622375151 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 45268+0+0 (secure 0 0 0) 0x7f7eb802d3f0 con 0x7f7ec8105380 2024-01-24T13:30:19.000 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.267+0000 7f7eaffff700 1 --2- 172.21.15.29:0/3622375151 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f7eb0038350 0x7f7eb003a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:19.000 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.267+0000 7f7eaffff700 1 -- 172.21.15.29:0/3622375151 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f7eb804c3a0 con 0x7f7ec8105380 2024-01-24T13:30:19.000 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.267+0000 7f7ec67fc700 1 --2- 172.21.15.29:0/3622375151 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f7eb0038350 0x7f7eb003a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:19.000 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.268+0000 7f7ecd6ad700 1 -- 172.21.15.29:0/3622375151 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ec8062370 con 0x7f7ec8105380 2024-01-24T13:30:19.001 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.268+0000 7f7ec67fc700 1 --2- 172.21.15.29:0/3622375151 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f7eb0038350 0x7f7eb003a810 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc006fd0 tx=0x7f7ebc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:19.001 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.273+0000 7f7eaffff700 1 -- 172.21.15.29:0/3622375151 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f7eb8010970 con 0x7f7ec8105380 2024-01-24T13:30:19.001 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.509+0000 7f7ecd6ad700 1 -- 172.21.15.29:0/3622375151 --> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f7ec8102b20 con 0x7f7eb0038350 2024-01-24T13:30:19.001 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.514+0000 7f7eaffff700 1 -- 172.21.15.29:0/3622375151 <== mgr.14120 v2:172.21.15.29:6800/1996568496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f7ec8102b20 con 0x7f7eb0038350 2024-01-24T13:30:19.001 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.516+0000 7f7ecd6ad700 1 -- 172.21.15.29:0/3622375151 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f7eb0038350 msgr2=0x7f7eb003a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:19.001 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.516+0000 7f7ecd6ad700 1 --2- 172.21.15.29:0/3622375151 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f7eb0038350 0x7f7eb003a810 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc006fd0 tx=0x7f7ebc006e40 comp rx=0 tx=0).stop 2024-01-24T13:30:19.001 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.516+0000 7f7ecd6ad700 1 -- 172.21.15.29:0/3622375151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ec8105380 msgr2=0x7f7ec819a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:19.001 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.516+0000 7f7ecd6ad700 1 --2- 172.21.15.29:0/3622375151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ec8105380 0x7f7ec819a130 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7eb8009130 tx=0x7f7eb8011ad0 comp rx=0 tx=0).stop 2024-01-24T13:30:19.001 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.516+0000 7f7ecd6ad700 1 -- 172.21.15.29:0/3622375151 shutdown_connections 2024-01-24T13:30:19.001 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.516+0000 7f7ecd6ad700 1 --2- 172.21.15.29:0/3622375151 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f7eb0038350 0x7f7eb003a810 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:19.001 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.516+0000 7f7ecd6ad700 1 --2- 172.21.15.29:0/3622375151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ec8105380 0x7f7ec819a130 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:19.001 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.516+0000 7f7ecd6ad700 1 -- 172.21.15.29:0/3622375151 >> 172.21.15.29:0/3622375151 conn(0x7f7ec8100940 msgr2=0x7f7ec80798c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:19.001 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.516+0000 7f7ecd6ad700 1 -- 172.21.15.29:0/3622375151 shutdown_connections 2024-01-24T13:30:19.001 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:18.516+0000 7f7ecd6ad700 1 -- 172.21.15.29:0/3622375151 wait complete. 2024-01-24T13:30:19.001 INFO:teuthology.orchestra.run.smithi029.stdout:Deploying unmanaged mgr service... 2024-01-24T13:30:19.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:19 smithi029 ceph-mon[40651]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:30:19.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:19 smithi029 ceph-mon[40651]: Saving service mon spec with placement count:5 2024-01-24T13:30:19.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:19 smithi029 ceph-mon[40651]: from='mgr.14120 172.21.15.29:0/656713355' entity='mgr.x' 2024-01-24T13:30:19.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:19 smithi029 ceph-mon[40651]: from='mgr.14120 172.21.15.29:0/656713355' entity='mgr.x' 2024-01-24T13:30:19.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:19 smithi029 ceph-mon[40651]: from='mgr.14120 172.21.15.29:0/656713355' entity='mgr.x' 2024-01-24T13:30:20.236 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-01-24T13:30:20.237 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.362+0000 7f1f661d9700 1 Processor -- start 2024-01-24T13:30:20.237 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.362+0000 7f1f661d9700 1 -- start start 2024-01-24T13:30:20.237 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.362+0000 7f1f661d9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f60104780 0x7f1f60104ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:20.237 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.362+0000 7f1f661d9700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f60105170 con 0x7f1f60104780 2024-01-24T13:30:20.237 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.362+0000 7f1f5f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f60104780 0x7f1f60104ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:20.237 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.362+0000 7f1f5f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f60104780 0x7f1f60104ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43946/0 (socket says 172.21.15.29:43946) 2024-01-24T13:30:20.237 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.362+0000 7f1f5f7fe700 1 -- 172.21.15.29:0/1731485179 learned_addr learned my addr 172.21.15.29:0/1731485179 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:20.237 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.363+0000 7f1f5f7fe700 1 -- 172.21.15.29:0/1731485179 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f60107710 con 0x7f1f60104780 2024-01-24T13:30:20.237 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.363+0000 7f1f5f7fe700 1 --2- 172.21.15.29:0/1731485179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f60104780 0x7f1f60104ba0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1f500098a0 tx=0x7f1f50009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f383f7bd6bef6133 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:20.237 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.363+0000 7f1f5e7fc700 1 -- 172.21.15.29:0/1731485179 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f50004160 con 0x7f1f60104780 2024-01-24T13:30:20.237 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.363+0000 7f1f5e7fc700 1 -- 172.21.15.29:0/1731485179 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1f500042c0 con 0x7f1f60104780 2024-01-24T13:30:20.238 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.363+0000 7f1f5e7fc700 1 -- 172.21.15.29:0/1731485179 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f500119b0 con 0x7f1f60104780 2024-01-24T13:30:20.238 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.364+0000 7f1f661d9700 1 -- 172.21.15.29:0/1731485179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f60104780 msgr2=0x7f1f60104ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:20.238 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.364+0000 7f1f661d9700 1 --2- 172.21.15.29:0/1731485179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f60104780 0x7f1f60104ba0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1f500098a0 tx=0x7f1f50009bb0 comp rx=0 tx=0).stop 2024-01-24T13:30:20.238 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.364+0000 7f1f661d9700 1 -- 172.21.15.29:0/1731485179 shutdown_connections 2024-01-24T13:30:20.238 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.364+0000 7f1f661d9700 1 --2- 172.21.15.29:0/1731485179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f60104780 0x7f1f60104ba0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:20.238 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.364+0000 7f1f661d9700 1 -- 172.21.15.29:0/1731485179 >> 172.21.15.29:0/1731485179 conn(0x7f1f601000d0 msgr2=0x7f1f601024f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:20.238 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.364+0000 7f1f661d9700 1 -- 172.21.15.29:0/1731485179 shutdown_connections 2024-01-24T13:30:20.238 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.364+0000 7f1f661d9700 1 -- 172.21.15.29:0/1731485179 wait complete. 2024-01-24T13:30:20.238 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.365+0000 7f1f661d9700 1 Processor -- start 2024-01-24T13:30:20.238 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.365+0000 7f1f661d9700 1 -- start start 2024-01-24T13:30:20.238 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.365+0000 7f1f661d9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f60104780 0x7f1f6019c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:20.238 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.365+0000 7f1f661d9700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6019c8a0 con 0x7f1f60104780 2024-01-24T13:30:20.238 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.365+0000 7f1f5f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f60104780 0x7f1f6019c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:20.238 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.365+0000 7f1f5f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f60104780 0x7f1f6019c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43952/0 (socket says 172.21.15.29:43952) 2024-01-24T13:30:20.239 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.365+0000 7f1f5f7fe700 1 -- 172.21.15.29:0/1666862144 learned_addr learned my addr 172.21.15.29:0/1666862144 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:20.239 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.366+0000 7f1f5f7fe700 1 -- 172.21.15.29:0/1666862144 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f50009550 con 0x7f1f60104780 2024-01-24T13:30:20.239 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.366+0000 7f1f5f7fe700 1 --2- 172.21.15.29:0/1666862144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f60104780 0x7f1f6019c360 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1f50006af0 tx=0x7f1f50011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:20.239 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.366+0000 7f1f5cff9700 1 -- 172.21.15.29:0/1666862144 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f50003a40 con 0x7f1f60104780 2024-01-24T13:30:20.239 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.366+0000 7f1f5cff9700 1 -- 172.21.15.29:0/1666862144 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1f50003ba0 con 0x7f1f60104780 2024-01-24T13:30:20.239 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.366+0000 7f1f661d9700 1 -- 172.21.15.29:0/1666862144 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f6019caa0 con 0x7f1f60104780 2024-01-24T13:30:20.239 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.366+0000 7f1f5cff9700 1 -- 172.21.15.29:0/1666862144 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f50019690 con 0x7f1f60104780 2024-01-24T13:30:20.239 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.366+0000 7f1f661d9700 1 -- 172.21.15.29:0/1666862144 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f6019cf40 con 0x7f1f60104780 2024-01-24T13:30:20.239 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.367+0000 7f1f5cff9700 1 -- 172.21.15.29:0/1666862144 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 45268+0+0 (secure 0 0 0) 0x7f1f50019900 con 0x7f1f60104780 2024-01-24T13:30:20.239 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.367+0000 7f1f661d9700 1 -- 172.21.15.29:0/1666862144 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f6010a900 con 0x7f1f60104780 2024-01-24T13:30:20.239 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.367+0000 7f1f5cff9700 1 --2- 172.21.15.29:0/1666862144 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f1f48038300 0x7f1f4803a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:20.239 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.367+0000 7f1f5cff9700 1 -- 172.21.15.29:0/1666862144 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f1f5004c100 con 0x7f1f60104780 2024-01-24T13:30:20.239 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.367+0000 7f1f5effd700 1 --2- 172.21.15.29:0/1666862144 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f1f48038300 0x7f1f4803a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:20.240 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.368+0000 7f1f5effd700 1 --2- 172.21.15.29:0/1666862144 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f1f48038300 0x7f1f4803a7c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1f54006fd0 tx=0x7f1f54006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:20.240 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.372+0000 7f1f5cff9700 1 -- 172.21.15.29:0/1666862144 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f1f50019e30 con 0x7f1f60104780 2024-01-24T13:30:20.240 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.609+0000 7f1f661d9700 1 -- 172.21.15.29:0/1666862144 --> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f1f6019fd20 con 0x7f1f48038300 2024-01-24T13:30:20.240 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.614+0000 7f1f5cff9700 1 -- 172.21.15.29:0/1666862144 <== mgr.14120 v2:172.21.15.29:6800/1996568496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f1f6019fd20 con 0x7f1f48038300 2024-01-24T13:30:20.240 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.616+0000 7f1f661d9700 1 -- 172.21.15.29:0/1666862144 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f1f48038300 msgr2=0x7f1f4803a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:20.240 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.616+0000 7f1f661d9700 1 --2- 172.21.15.29:0/1666862144 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f1f48038300 0x7f1f4803a7c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1f54006fd0 tx=0x7f1f54006e40 comp rx=0 tx=0).stop 2024-01-24T13:30:20.240 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.616+0000 7f1f661d9700 1 -- 172.21.15.29:0/1666862144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f60104780 msgr2=0x7f1f6019c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:20.240 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.616+0000 7f1f661d9700 1 --2- 172.21.15.29:0/1666862144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f60104780 0x7f1f6019c360 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1f50006af0 tx=0x7f1f50011a10 comp rx=0 tx=0).stop 2024-01-24T13:30:20.240 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.616+0000 7f1f661d9700 1 -- 172.21.15.29:0/1666862144 shutdown_connections 2024-01-24T13:30:20.240 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.616+0000 7f1f661d9700 1 --2- 172.21.15.29:0/1666862144 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f1f48038300 0x7f1f4803a7c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:20.240 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.616+0000 7f1f661d9700 1 --2- 172.21.15.29:0/1666862144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f60104780 0x7f1f6019c360 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:20.240 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.616+0000 7f1f661d9700 1 -- 172.21.15.29:0/1666862144 >> 172.21.15.29:0/1666862144 conn(0x7f1f601000d0 msgr2=0x7f1f60100e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:20.240 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.616+0000 7f1f661d9700 1 -- 172.21.15.29:0/1666862144 shutdown_connections 2024-01-24T13:30:20.241 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:19.616+0000 7f1f661d9700 1 -- 172.21.15.29:0/1666862144 wait complete. 2024-01-24T13:30:20.894 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:20 smithi029 ceph-mon[40651]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:30:20.894 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:20 smithi029 ceph-mon[40651]: Saving service mgr spec with placement count:2 2024-01-24T13:30:20.894 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:20 smithi029 ceph-mon[40651]: from='mgr.14120 172.21.15.29:0/656713355' entity='mgr.x' 2024-01-24T13:30:20.894 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:20 smithi029 ceph-mon[40651]: from='mgr.14120 172.21.15.29:0/656713355' entity='mgr.x' 2024-01-24T13:30:21.237 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.633+0000 7f1f68c7c700 1 Processor -- start 2024-01-24T13:30:21.237 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.633+0000 7f1f68c7c700 1 -- start start 2024-01-24T13:30:21.237 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.634+0000 7f1f68c7c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f641064a0 0x7f1f641068c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:21.237 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.634+0000 7f1f68c7c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f64106e90 con 0x7f1f641064a0 2024-01-24T13:30:21.238 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.634+0000 7f1f6259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f641064a0 0x7f1f641068c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:21.238 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.634+0000 7f1f6259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f641064a0 0x7f1f641068c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43960/0 (socket says 172.21.15.29:43960) 2024-01-24T13:30:21.238 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.634+0000 7f1f6259c700 1 -- 172.21.15.29:0/1238277178 learned_addr learned my addr 172.21.15.29:0/1238277178 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:21.238 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.634+0000 7f1f6259c700 1 -- 172.21.15.29:0/1238277178 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f641076a0 con 0x7f1f641064a0 2024-01-24T13:30:21.238 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.635+0000 7f1f6259c700 1 --2- 172.21.15.29:0/1238277178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f641064a0 0x7f1f641068c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1f540098a0 tx=0x7f1f54009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=512c12d284608235 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:21.238 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.635+0000 7f1f6159a700 1 -- 172.21.15.29:0/1238277178 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f54004160 con 0x7f1f641064a0 2024-01-24T13:30:21.238 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.635+0000 7f1f6159a700 1 -- 172.21.15.29:0/1238277178 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1f540042c0 con 0x7f1f641064a0 2024-01-24T13:30:21.238 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.635+0000 7f1f6159a700 1 -- 172.21.15.29:0/1238277178 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f540119b0 con 0x7f1f641064a0 2024-01-24T13:30:21.238 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.636+0000 7f1f68c7c700 1 -- 172.21.15.29:0/1238277178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f641064a0 msgr2=0x7f1f641068c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:21.238 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.636+0000 7f1f68c7c700 1 --2- 172.21.15.29:0/1238277178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f641064a0 0x7f1f641068c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1f540098a0 tx=0x7f1f54009bb0 comp rx=0 tx=0).stop 2024-01-24T13:30:21.239 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.636+0000 7f1f68c7c700 1 -- 172.21.15.29:0/1238277178 shutdown_connections 2024-01-24T13:30:21.239 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.636+0000 7f1f68c7c700 1 --2- 172.21.15.29:0/1238277178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f641064a0 0x7f1f641068c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:21.239 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.636+0000 7f1f68c7c700 1 -- 172.21.15.29:0/1238277178 >> 172.21.15.29:0/1238277178 conn(0x7f1f64101a60 msgr2=0x7f1f64103e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:21.239 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.636+0000 7f1f68c7c700 1 -- 172.21.15.29:0/1238277178 shutdown_connections 2024-01-24T13:30:21.239 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.636+0000 7f1f68c7c700 1 -- 172.21.15.29:0/1238277178 wait complete. 2024-01-24T13:30:21.239 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.637+0000 7f1f68c7c700 1 Processor -- start 2024-01-24T13:30:21.239 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.637+0000 7f1f68c7c700 1 -- start start 2024-01-24T13:30:21.239 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.637+0000 7f1f68c7c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f641064a0 0x7f1f6419a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:21.239 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.637+0000 7f1f68c7c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6419a620 con 0x7f1f641064a0 2024-01-24T13:30:21.239 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.637+0000 7f1f6259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f641064a0 0x7f1f6419a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:21.240 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.637+0000 7f1f6259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f641064a0 0x7f1f6419a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43966/0 (socket says 172.21.15.29:43966) 2024-01-24T13:30:21.240 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.637+0000 7f1f6259c700 1 -- 172.21.15.29:0/1680686300 learned_addr learned my addr 172.21.15.29:0/1680686300 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:21.240 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.638+0000 7f1f6259c700 1 -- 172.21.15.29:0/1680686300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f54009550 con 0x7f1f641064a0 2024-01-24T13:30:21.240 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.638+0000 7f1f6259c700 1 --2- 172.21.15.29:0/1680686300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f641064a0 0x7f1f6419a0e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1f54000c00 tx=0x7f1f54011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:21.240 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.638+0000 7f1f537fe700 1 -- 172.21.15.29:0/1680686300 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f540039f0 con 0x7f1f641064a0 2024-01-24T13:30:21.240 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.638+0000 7f1f537fe700 1 -- 172.21.15.29:0/1680686300 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1f54003b50 con 0x7f1f641064a0 2024-01-24T13:30:21.240 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.638+0000 7f1f68c7c700 1 -- 172.21.15.29:0/1680686300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f6419a820 con 0x7f1f641064a0 2024-01-24T13:30:21.240 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.638+0000 7f1f537fe700 1 -- 172.21.15.29:0/1680686300 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f5401a780 con 0x7f1f641064a0 2024-01-24T13:30:21.240 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.638+0000 7f1f68c7c700 1 -- 172.21.15.29:0/1680686300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f6419acc0 con 0x7f1f641064a0 2024-01-24T13:30:21.240 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.639+0000 7f1f537fe700 1 -- 172.21.15.29:0/1680686300 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 45268+0+0 (secure 0 0 0) 0x7f1f5401a9f0 con 0x7f1f641064a0 2024-01-24T13:30:21.240 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.639+0000 7f1f537fe700 1 --2- 172.21.15.29:0/1680686300 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f1f4c038350 0x7f1f4c03a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:21.240 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.639+0000 7f1f537fe700 1 -- 172.21.15.29:0/1680686300 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f1f54024440 con 0x7f1f641064a0 2024-01-24T13:30:21.241 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.639+0000 7f1f61d9b700 1 --2- 172.21.15.29:0/1680686300 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f1f4c038350 0x7f1f4c03a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:21.241 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.639+0000 7f1f68c7c700 1 -- 172.21.15.29:0/1680686300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f64062370 con 0x7f1f641064a0 2024-01-24T13:30:21.241 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.640+0000 7f1f61d9b700 1 --2- 172.21.15.29:0/1680686300 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f1f4c038350 0x7f1f4c03a810 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1f58006fd0 tx=0x7f1f58006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:21.241 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.645+0000 7f1f537fe700 1 -- 172.21.15.29:0/1680686300 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f1f5401aca0 con 0x7f1f641064a0 2024-01-24T13:30:21.241 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.859+0000 7f1f68c7c700 1 -- 172.21.15.29:0/1680686300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f1f6419d680 con 0x7f1f641064a0 2024-01-24T13:30:21.241 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.862+0000 7f1f537fe700 1 -- 172.21.15.29:0/1680686300 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f1f5404b0d0 con 0x7f1f641064a0 2024-01-24T13:30:21.241 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.865+0000 7f1f68c7c700 1 -- 172.21.15.29:0/1680686300 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f1f4c038350 msgr2=0x7f1f4c03a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:21.241 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.865+0000 7f1f68c7c700 1 --2- 172.21.15.29:0/1680686300 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f1f4c038350 0x7f1f4c03a810 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1f58006fd0 tx=0x7f1f58006e40 comp rx=0 tx=0).stop 2024-01-24T13:30:21.241 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.865+0000 7f1f68c7c700 1 -- 172.21.15.29:0/1680686300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f641064a0 msgr2=0x7f1f6419a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:21.241 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.865+0000 7f1f68c7c700 1 --2- 172.21.15.29:0/1680686300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f641064a0 0x7f1f6419a0e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1f54000c00 tx=0x7f1f54011a30 comp rx=0 tx=0).stop 2024-01-24T13:30:21.241 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.865+0000 7f1f68c7c700 1 -- 172.21.15.29:0/1680686300 shutdown_connections 2024-01-24T13:30:21.241 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.865+0000 7f1f68c7c700 1 --2- 172.21.15.29:0/1680686300 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f1f4c038350 0x7f1f4c03a810 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:21.241 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.865+0000 7f1f68c7c700 1 --2- 172.21.15.29:0/1680686300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f641064a0 0x7f1f6419a0e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:21.241 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.865+0000 7f1f68c7c700 1 -- 172.21.15.29:0/1680686300 >> 172.21.15.29:0/1680686300 conn(0x7f1f64101a60 msgr2=0x7f1f641027d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:21.241 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.865+0000 7f1f68c7c700 1 -- 172.21.15.29:0/1680686300 shutdown_connections 2024-01-24T13:30:21.241 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:20.865+0000 7f1f68c7c700 1 -- 172.21.15.29:0/1680686300 wait complete. 2024-01-24T13:30:21.938 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:21 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/1680686300' entity='client.admin' 2024-01-24T13:30:21.939 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:21 smithi029 ceph-mon[40651]: from='mgr.14120 172.21.15.29:0/656713355' entity='mgr.x' 2024-01-24T13:30:21.939 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:21 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/106853332' entity='client.admin' 2024-01-24T13:30:22.371 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.617+0000 7fbdb1cae700 1 Processor -- start 2024-01-24T13:30:22.372 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.617+0000 7fbdb1cae700 1 -- start start 2024-01-24T13:30:22.372 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.618+0000 7fbdb1cae700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdac106f70 0x7fbdac109360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:22.372 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.618+0000 7fbdb1cae700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdac079e10 con 0x7fbdac106f70 2024-01-24T13:30:22.372 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.618+0000 7fbdab7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdac106f70 0x7fbdac109360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:22.372 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.618+0000 7fbdab7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdac106f70 0x7fbdac109360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39012/0 (socket says 172.21.15.29:39012) 2024-01-24T13:30:22.372 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.618+0000 7fbdab7fe700 1 -- 172.21.15.29:0/3986606895 learned_addr learned my addr 172.21.15.29:0/3986606895 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:22.372 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.618+0000 7fbdab7fe700 1 -- 172.21.15.29:0/3986606895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdac1098a0 con 0x7fbdac106f70 2024-01-24T13:30:22.372 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.618+0000 7fbdab7fe700 1 --2- 172.21.15.29:0/3986606895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdac106f70 0x7fbdac109360 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbda4009e80 tx=0x7fbda400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd98a320899c2c46 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:22.373 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.619+0000 7fbdaa7fc700 1 -- 172.21.15.29:0/3986606895 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbda400e070 con 0x7fbdac106f70 2024-01-24T13:30:22.373 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.619+0000 7fbdaa7fc700 1 -- 172.21.15.29:0/3986606895 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbda400b8c0 con 0x7fbdac106f70 2024-01-24T13:30:22.373 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.619+0000 7fbdaa7fc700 1 -- 172.21.15.29:0/3986606895 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbda40043e0 con 0x7fbdac106f70 2024-01-24T13:30:22.373 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.619+0000 7fbdb1cae700 1 -- 172.21.15.29:0/3986606895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdac106f70 msgr2=0x7fbdac109360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:22.373 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.619+0000 7fbdb1cae700 1 --2- 172.21.15.29:0/3986606895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdac106f70 0x7fbdac109360 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbda4009e80 tx=0x7fbda400b330 comp rx=0 tx=0).stop 2024-01-24T13:30:22.373 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.620+0000 7fbdb1cae700 1 -- 172.21.15.29:0/3986606895 shutdown_connections 2024-01-24T13:30:22.373 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.620+0000 7fbdb1cae700 1 --2- 172.21.15.29:0/3986606895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdac106f70 0x7fbdac109360 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:22.373 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.620+0000 7fbdb1cae700 1 -- 172.21.15.29:0/3986606895 >> 172.21.15.29:0/3986606895 conn(0x7fbdac100940 msgr2=0x7fbdac102d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:22.373 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.620+0000 7fbdb1cae700 1 -- 172.21.15.29:0/3986606895 shutdown_connections 2024-01-24T13:30:22.373 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.620+0000 7fbdb1cae700 1 -- 172.21.15.29:0/3986606895 wait complete. 2024-01-24T13:30:22.373 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.620+0000 7fbdb1cae700 1 Processor -- start 2024-01-24T13:30:22.373 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.621+0000 7fbdb1cae700 1 -- start start 2024-01-24T13:30:22.373 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.621+0000 7fbdb1cae700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdac106f70 0x7fbdac19c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:22.373 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.621+0000 7fbdb1cae700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdac19c800 con 0x7fbdac106f70 2024-01-24T13:30:22.374 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.621+0000 7fbdab7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdac106f70 0x7fbdac19c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:22.374 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.621+0000 7fbdab7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdac106f70 0x7fbdac19c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39016/0 (socket says 172.21.15.29:39016) 2024-01-24T13:30:22.374 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.621+0000 7fbdab7fe700 1 -- 172.21.15.29:0/106853332 learned_addr learned my addr 172.21.15.29:0/106853332 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:22.374 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.622+0000 7fbdab7fe700 1 -- 172.21.15.29:0/106853332 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda4009b30 con 0x7fbdac106f70 2024-01-24T13:30:22.374 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.622+0000 7fbdab7fe700 1 --2- 172.21.15.29:0/106853332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdac106f70 0x7fbdac19c2c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbda4003fb0 tx=0x7fbda4003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:22.374 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.622+0000 7fbda8ff9700 1 -- 172.21.15.29:0/106853332 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbda400e040 con 0x7fbdac106f70 2024-01-24T13:30:22.374 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.622+0000 7fbdb1cae700 1 -- 172.21.15.29:0/106853332 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdac19ca00 con 0x7fbdac106f70 2024-01-24T13:30:22.374 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.622+0000 7fbda8ff9700 1 -- 172.21.15.29:0/106853332 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbda400bd70 con 0x7fbdac106f70 2024-01-24T13:30:22.374 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.622+0000 7fbda8ff9700 1 -- 172.21.15.29:0/106853332 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbda401c620 con 0x7fbdac106f70 2024-01-24T13:30:22.374 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.622+0000 7fbdb1cae700 1 -- 172.21.15.29:0/106853332 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdac19cf00 con 0x7fbdac106f70 2024-01-24T13:30:22.375 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.623+0000 7fbda8ff9700 1 -- 172.21.15.29:0/106853332 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 45268+0+0 (secure 0 0 0) 0x7fbda401c8e0 con 0x7fbdac106f70 2024-01-24T13:30:22.375 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.624+0000 7fbdb1cae700 1 -- 172.21.15.29:0/106853332 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdac10a630 con 0x7fbdac106f70 2024-01-24T13:30:22.375 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.624+0000 7fbda8ff9700 1 --2- 172.21.15.29:0/106853332 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7fbd980382e0 0x7fbd9803a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:22.375 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.624+0000 7fbda8ff9700 1 -- 172.21.15.29:0/106853332 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fbda404c080 con 0x7fbdac106f70 2024-01-24T13:30:22.375 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.624+0000 7fbdaaffd700 1 --2- 172.21.15.29:0/106853332 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7fbd980382e0 0x7fbd9803a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:22.375 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.624+0000 7fbdaaffd700 1 --2- 172.21.15.29:0/106853332 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7fbd980382e0 0x7fbd9803a7a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fbd94006fd0 tx=0x7fbd94006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:22.375 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.630+0000 7fbda8ff9700 1 -- 172.21.15.29:0/106853332 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fbda4011980 con 0x7fbdac106f70 2024-01-24T13:30:22.375 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.847+0000 7fbdb1cae700 1 -- 172.21.15.29:0/106853332 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fbdac10a850 con 0x7fbdac106f70 2024-01-24T13:30:22.375 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.851+0000 7fbda8ff9700 1 -- 172.21.15.29:0/106853332 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fbda4017070 con 0x7fbdac106f70 2024-01-24T13:30:22.375 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.853+0000 7fbdb1cae700 1 -- 172.21.15.29:0/106853332 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7fbd980382e0 msgr2=0x7fbd9803a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:22.375 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.853+0000 7fbdb1cae700 1 --2- 172.21.15.29:0/106853332 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7fbd980382e0 0x7fbd9803a7a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fbd94006fd0 tx=0x7fbd94006e40 comp rx=0 tx=0).stop 2024-01-24T13:30:22.375 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.853+0000 7fbdb1cae700 1 -- 172.21.15.29:0/106853332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdac106f70 msgr2=0x7fbdac19c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:22.375 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.853+0000 7fbdb1cae700 1 --2- 172.21.15.29:0/106853332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdac106f70 0x7fbdac19c2c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbda4003fb0 tx=0x7fbda4003d00 comp rx=0 tx=0).stop 2024-01-24T13:30:22.376 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.853+0000 7fbdb1cae700 1 -- 172.21.15.29:0/106853332 shutdown_connections 2024-01-24T13:30:22.376 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.853+0000 7fbdb1cae700 1 --2- 172.21.15.29:0/106853332 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7fbd980382e0 0x7fbd9803a7a0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:22.376 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.853+0000 7fbdb1cae700 1 --2- 172.21.15.29:0/106853332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdac106f70 0x7fbdac19c2c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:22.376 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.853+0000 7fbdb1cae700 1 -- 172.21.15.29:0/106853332 >> 172.21.15.29:0/106853332 conn(0x7fbdac100940 msgr2=0x7fbdac101680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:22.376 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.854+0000 7fbdb1cae700 1 -- 172.21.15.29:0/106853332 shutdown_connections 2024-01-24T13:30:22.376 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:21.854+0000 7fbdb1cae700 1 -- 172.21.15.29:0/106853332 wait complete. 2024-01-24T13:30:22.376 INFO:teuthology.orchestra.run.smithi029.stdout:Enabling the dashboard module... 2024-01-24T13:30:23.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:23 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/2483340248' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-01-24T13:30:23.417 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:23 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: ignoring --setuser ceph since I am not root 2024-01-24T13:30:23.417 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:23 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: ignoring --setgroup ceph since I am not root 2024-01-24T13:30:23.449 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:22.771+0000 7f1bec932700 1 Processor -- start 2024-01-24T13:30:23.449 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:22.772+0000 7f1bec932700 1 -- start start 2024-01-24T13:30:23.449 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:22.772+0000 7f1bec932700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1be8105380 0x7f1be81057a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:23.449 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:22.772+0000 7f1bec932700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be8105d70 con 0x7f1be8105380 2024-01-24T13:30:23.449 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:22.772+0000 7f1be659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1be8105380 0x7f1be81057a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:23.449 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:22.772+0000 7f1be659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1be8105380 0x7f1be81057a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39028/0 (socket says 172.21.15.29:39028) 2024-01-24T13:30:23.449 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:22.772+0000 7f1be659c700 1 -- 172.21.15.29:0/528049884 learned_addr learned my addr 172.21.15.29:0/528049884 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:23.449 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:22.773+0000 7f1be659c700 1 -- 172.21.15.29:0/528049884 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1be8106580 con 0x7f1be8105380 2024-01-24T13:30:23.449 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:22.773+0000 7f1be659c700 1 --2- 172.21.15.29:0/528049884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1be8105380 0x7f1be81057a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1bd80098a0 tx=0x7f1bd8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86eb0b2cc6ab564c server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:23.449 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:22.773+0000 7f1be559a700 1 -- 172.21.15.29:0/528049884 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1bd8004160 con 0x7f1be8105380 2024-01-24T13:30:23.449 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:22.773+0000 7f1be559a700 1 -- 172.21.15.29:0/528049884 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1bd80042c0 con 0x7f1be8105380 2024-01-24T13:30:23.449 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:22.773+0000 7f1be559a700 1 -- 172.21.15.29:0/528049884 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1bd80119b0 con 0x7f1be8105380 2024-01-24T13:30:23.450 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:22.774+0000 7f1bec932700 1 -- 172.21.15.29:0/528049884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1be8105380 msgr2=0x7f1be81057a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:23.450 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:22.774+0000 7f1bec932700 1 --2- 172.21.15.29:0/528049884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1be8105380 0x7f1be81057a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1bd80098a0 tx=0x7f1bd8009bb0 comp rx=0 tx=0).stop 2024-01-24T13:30:23.450 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:22.774+0000 7f1bec932700 1 -- 172.21.15.29:0/528049884 shutdown_connections 2024-01-24T13:30:23.450 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:22.774+0000 7f1bec932700 1 --2- 172.21.15.29:0/528049884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1be8105380 0x7f1be81057a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:23.450 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:22.774+0000 7f1bec932700 1 -- 172.21.15.29:0/528049884 >> 172.21.15.29:0/528049884 conn(0x7f1be8100940 msgr2=0x7f1be8102d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:23.450 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:22.774+0000 7f1bec932700 1 -- 172.21.15.29:0/528049884 shutdown_connections 2024-01-24T13:30:23.450 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:22.774+0000 7f1bec932700 1 -- 172.21.15.29:0/528049884 wait complete. 2024-01-24T13:30:23.450 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:22.775+0000 7f1bec932700 1 Processor -- start 2024-01-24T13:30:23.450 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:22.775+0000 7f1bec932700 1 -- start start 2024-01-24T13:30:23.450 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:22.775+0000 7f1bec932700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1be8105380 0x7f1be819a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:23.450 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:22.775+0000 7f1bec932700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be819a680 con 0x7f1be8105380 2024-01-24T13:30:23.450 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:22.775+0000 7f1be659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1be8105380 0x7f1be819a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:23.450 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:22.775+0000 7f1be659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1be8105380 0x7f1be819a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39040/0 (socket says 172.21.15.29:39040) 2024-01-24T13:30:23.450 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:22.775+0000 7f1be659c700 1 -- 172.21.15.29:0/2483340248 learned_addr learned my addr 172.21.15.29:0/2483340248 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:23.451 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:22.775+0000 7f1be659c700 1 -- 172.21.15.29:0/2483340248 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd8009550 con 0x7f1be8105380 2024-01-24T13:30:23.451 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:22.776+0000 7f1be659c700 1 --2- 172.21.15.29:0/2483340248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1be8105380 0x7f1be819a140 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1bd8003e60 tx=0x7f1bd8011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:23.451 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:22.776+0000 7f1bd77fe700 1 -- 172.21.15.29:0/2483340248 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1bd8003a60 con 0x7f1be8105380 2024-01-24T13:30:23.451 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:22.776+0000 7f1bec932700 1 -- 172.21.15.29:0/2483340248 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1be819a880 con 0x7f1be8105380 2024-01-24T13:30:23.451 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:22.776+0000 7f1bd77fe700 1 -- 172.21.15.29:0/2483340248 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1bd8003bc0 con 0x7f1be8105380 2024-01-24T13:30:23.451 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:22.776+0000 7f1bd77fe700 1 -- 172.21.15.29:0/2483340248 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1bd80196c0 con 0x7f1be8105380 2024-01-24T13:30:23.451 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:22.776+0000 7f1bec932700 1 -- 172.21.15.29:0/2483340248 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1be819ad20 con 0x7f1be8105380 2024-01-24T13:30:23.451 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:22.777+0000 7f1bd77fe700 1 -- 172.21.15.29:0/2483340248 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) v1 ==== 45268+0+0 (secure 0 0 0) 0x7f1bd8019980 con 0x7f1be8105380 2024-01-24T13:30:23.451 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:22.777+0000 7f1bec932700 1 -- 172.21.15.29:0/2483340248 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1be8193e80 con 0x7f1be8105380 2024-01-24T13:30:23.451 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:22.777+0000 7f1bd77fe700 1 --2- 172.21.15.29:0/2483340248 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f1bd00382e0 0x7f1bd003a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:23.451 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:22.777+0000 7f1bd77fe700 1 -- 172.21.15.29:0/2483340248 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f1bd804c6b0 con 0x7f1be8105380 2024-01-24T13:30:23.451 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:22.777+0000 7f1be5d9b700 1 --2- 172.21.15.29:0/2483340248 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f1bd00382e0 0x7f1bd003a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:23.451 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:22.778+0000 7f1be5d9b700 1 --2- 172.21.15.29:0/2483340248 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f1bd00382e0 0x7f1bd003a7a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc006fd0 tx=0x7f1bdc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:23.452 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:22.782+0000 7f1bd77fe700 1 -- 172.21.15.29:0/2483340248 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f1bd8010b70 con 0x7f1be8105380 2024-01-24T13:30:23.452 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:23.046+0000 7f1bec932700 1 -- 172.21.15.29:0/2483340248 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f1be8061950 con 0x7f1be8105380 2024-01-24T13:30:23.452 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:23.099+0000 7f1bd77fe700 1 -- 172.21.15.29:0/2483340248 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 9) v1 ==== 45281+0+0 (secure 0 0 0) 0x7f1bd800fd70 con 0x7f1be8105380 2024-01-24T13:30:23.452 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:23.100+0000 7f1bd77fe700 1 -- 172.21.15.29:0/2483340248 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f1bd801ab60 con 0x7f1be8105380 2024-01-24T13:30:23.452 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:23.103+0000 7f1bec932700 1 -- 172.21.15.29:0/2483340248 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f1bd00382e0 msgr2=0x7f1bd003a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:23.452 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:23.103+0000 7f1bec932700 1 --2- 172.21.15.29:0/2483340248 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f1bd00382e0 0x7f1bd003a7a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc006fd0 tx=0x7f1bdc006e40 comp rx=0 tx=0).stop 2024-01-24T13:30:23.452 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:23.103+0000 7f1bec932700 1 -- 172.21.15.29:0/2483340248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1be8105380 msgr2=0x7f1be819a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:23.452 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:23.103+0000 7f1bec932700 1 --2- 172.21.15.29:0/2483340248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1be8105380 0x7f1be819a140 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1bd8003e60 tx=0x7f1bd8011a30 comp rx=0 tx=0).stop 2024-01-24T13:30:23.452 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:23.103+0000 7f1bec932700 1 -- 172.21.15.29:0/2483340248 shutdown_connections 2024-01-24T13:30:23.452 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:23.103+0000 7f1bec932700 1 --2- 172.21.15.29:0/2483340248 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f1bd00382e0 0x7f1bd003a7a0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:23.452 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:23.103+0000 7f1bec932700 1 --2- 172.21.15.29:0/2483340248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1be8105380 0x7f1be819a140 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:23.452 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:23.103+0000 7f1bec932700 1 -- 172.21.15.29:0/2483340248 >> 172.21.15.29:0/2483340248 conn(0x7f1be8100940 msgr2=0x7f1be81015b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:23.452 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:23.103+0000 7f1bec932700 1 -- 172.21.15.29:0/2483340248 shutdown_connections 2024-01-24T13:30:23.452 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:23.103+0000 7f1bec932700 1 -- 172.21.15.29:0/2483340248 wait complete. 2024-01-24T13:30:23.720 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:23 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:23.614+0000 7effe9b79200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-24T13:30:24.099 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:23 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:23.720+0000 7effe9b79200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-24T13:30:24.099 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:23 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:23.819+0000 7effe9b79200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-24T13:30:24.099 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:23 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:23.913+0000 7effe9b79200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-24T13:30:24.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:24 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/2483340248' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-01-24T13:30:24.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:24 smithi029 ceph-mon[40651]: mgrmap e9: x(active, since 14s) 2024-01-24T13:30:24.849 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:24 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:24.653+0000 7effe9b79200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-24T13:30:25.332 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:25 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/30658381' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-24T13:30:25.333 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:24 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:24.929+0000 7effe9b79200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-24T13:30:25.333 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:25 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:25.028+0000 7effe9b79200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-24T13:30:25.333 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:25 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:25.151+0000 7effe9b79200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-24T13:30:25.333 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:25 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:25.332+0000 7effe9b79200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-24T13:30:25.599 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:25 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:25.477+0000 7effe9b79200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-24T13:30:25.599 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:25 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:25.573+0000 7effe9b79200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-24T13:30:26.298 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:25 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:25.998+0000 7effe9b79200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-24T13:30:26.599 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:26 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:26.298+0000 7effe9b79200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-24T13:30:26.812 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-01-24T13:30:26.812 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-01-24T13:30:26.812 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": true, 2024-01-24T13:30:26.812 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-01-24T13:30:26.812 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-01-24T13:30:26.812 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-01-24T13:30:26.812 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.194+0000 7f67a3071700 1 Processor -- start 2024-01-24T13:30:26.812 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.194+0000 7f67a3071700 1 -- start start 2024-01-24T13:30:26.812 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.194+0000 7f67a3071700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f679c106f50 0x7f679c109340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:26.813 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.194+0000 7f67a3071700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c079e50 con 0x7f679c106f50 2024-01-24T13:30:26.813 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.194+0000 7f67a0e0d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f679c106f50 0x7f679c109340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:26.813 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.194+0000 7f67a0e0d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f679c106f50 0x7f679c109340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39074/0 (socket says 172.21.15.29:39074) 2024-01-24T13:30:26.813 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.194+0000 7f67a0e0d700 1 -- 172.21.15.29:0/1093898196 learned_addr learned my addr 172.21.15.29:0/1093898196 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:26.813 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.195+0000 7f67a0e0d700 1 -- 172.21.15.29:0/1093898196 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f679c109880 con 0x7f679c106f50 2024-01-24T13:30:26.813 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.195+0000 7f67a0e0d700 1 --2- 172.21.15.29:0/1093898196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f679c106f50 0x7f679c109340 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f67900098a0 tx=0x7f6790009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdc14fcec5baceab server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:26.813 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.195+0000 7f679b7fe700 1 -- 172.21.15.29:0/1093898196 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6790004160 con 0x7f679c106f50 2024-01-24T13:30:26.813 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.195+0000 7f679b7fe700 1 -- 172.21.15.29:0/1093898196 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f67900042c0 con 0x7f679c106f50 2024-01-24T13:30:26.813 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.195+0000 7f679b7fe700 1 -- 172.21.15.29:0/1093898196 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f67900119b0 con 0x7f679c106f50 2024-01-24T13:30:26.813 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.196+0000 7f67a3071700 1 -- 172.21.15.29:0/1093898196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f679c106f50 msgr2=0x7f679c109340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:26.813 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.196+0000 7f67a3071700 1 --2- 172.21.15.29:0/1093898196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f679c106f50 0x7f679c109340 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f67900098a0 tx=0x7f6790009bb0 comp rx=0 tx=0).stop 2024-01-24T13:30:26.813 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.196+0000 7f67a3071700 1 -- 172.21.15.29:0/1093898196 shutdown_connections 2024-01-24T13:30:26.813 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.196+0000 7f67a3071700 1 --2- 172.21.15.29:0/1093898196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f679c106f50 0x7f679c109340 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:26.813 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.196+0000 7f67a3071700 1 -- 172.21.15.29:0/1093898196 >> 172.21.15.29:0/1093898196 conn(0x7f679c100980 msgr2=0x7f679c102da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:26.814 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.196+0000 7f67a3071700 1 -- 172.21.15.29:0/1093898196 shutdown_connections 2024-01-24T13:30:26.814 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.196+0000 7f67a3071700 1 -- 172.21.15.29:0/1093898196 wait complete. 2024-01-24T13:30:26.814 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.196+0000 7f67a3071700 1 Processor -- start 2024-01-24T13:30:26.814 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.196+0000 7f67a3071700 1 -- start start 2024-01-24T13:30:26.814 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.197+0000 7f67a3071700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f679c106f50 0x7f679c19c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:26.814 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.197+0000 7f67a3071700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c19c7f0 con 0x7f679c106f50 2024-01-24T13:30:26.814 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.197+0000 7f67a0e0d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f679c106f50 0x7f679c19c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:26.814 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.197+0000 7f67a0e0d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f679c106f50 0x7f679c19c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39088/0 (socket says 172.21.15.29:39088) 2024-01-24T13:30:26.814 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.197+0000 7f67a0e0d700 1 -- 172.21.15.29:0/30658381 learned_addr learned my addr 172.21.15.29:0/30658381 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:26.814 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.197+0000 7f67a0e0d700 1 -- 172.21.15.29:0/30658381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6790009550 con 0x7f679c106f50 2024-01-24T13:30:26.814 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.197+0000 7f67a0e0d700 1 --2- 172.21.15.29:0/30658381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f679c106f50 0x7f679c19c2b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6790011980 tx=0x7f6790011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:26.814 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.197+0000 7f6799ffb700 1 -- 172.21.15.29:0/30658381 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6790003a60 con 0x7f679c106f50 2024-01-24T13:30:26.814 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.197+0000 7f6799ffb700 1 -- 172.21.15.29:0/30658381 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6790003bc0 con 0x7f679c106f50 2024-01-24T13:30:26.814 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.197+0000 7f67a3071700 1 -- 172.21.15.29:0/30658381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f679c19c9f0 con 0x7f679c106f50 2024-01-24T13:30:26.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.197+0000 7f6799ffb700 1 -- 172.21.15.29:0/30658381 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f679001a670 con 0x7f679c106f50 2024-01-24T13:30:26.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.198+0000 7f67a3071700 1 -- 172.21.15.29:0/30658381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f679c19ce90 con 0x7f679c106f50 2024-01-24T13:30:26.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.198+0000 7f6799ffb700 1 -- 172.21.15.29:0/30658381 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 9) v1 ==== 45281+0+0 (secure 0 0 0) 0x7f679001a8e0 con 0x7f679c106f50 2024-01-24T13:30:26.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.199+0000 7f6799ffb700 1 --2- 172.21.15.29:0/30658381 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f678c0383a0 0x7f678c03a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:26.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.199+0000 7f6799ffb700 1 -- 172.21.15.29:0/30658381 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f679004c500 con 0x7f679c106f50 2024-01-24T13:30:26.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.199+0000 7f679bfff700 1 -- 172.21.15.29:0/30658381 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f678c0383a0 msgr2=0x7f678c03a860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/1996568496 2024-01-24T13:30:26.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.199+0000 7f679bfff700 1 --2- 172.21.15.29:0/30658381 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f678c0383a0 0x7f678c03a860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-24T13:30:26.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.199+0000 7f67a3071700 1 -- 172.21.15.29:0/30658381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f679c062370 con 0x7f679c106f50 2024-01-24T13:30:26.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.204+0000 7f6799ffb700 1 -- 172.21.15.29:0/30658381 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f679001abe0 con 0x7f679c106f50 2024-01-24T13:30:26.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.400+0000 7f679bfff700 1 -- 172.21.15.29:0/30658381 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f678c0383a0 msgr2=0x7f678c03a860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/1996568496 2024-01-24T13:30:26.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.400+0000 7f679bfff700 1 --2- 172.21.15.29:0/30658381 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f678c0383a0 0x7f678c03a860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-24T13:30:26.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.551+0000 7f67a3071700 1 -- 172.21.15.29:0/30658381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f679c19f940 con 0x7f679c106f50 2024-01-24T13:30:26.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.551+0000 7f6799ffb700 1 -- 172.21.15.29:0/30658381 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7f6790018350 con 0x7f679c106f50 2024-01-24T13:30:26.816 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.555+0000 7f67a3071700 1 -- 172.21.15.29:0/30658381 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f678c0383a0 msgr2=0x7f678c03a860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-24T13:30:26.816 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.555+0000 7f67a3071700 1 --2- 172.21.15.29:0/30658381 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f678c0383a0 0x7f678c03a860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:26.816 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.555+0000 7f67a3071700 1 -- 172.21.15.29:0/30658381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f679c106f50 msgr2=0x7f679c19c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:26.816 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.555+0000 7f67a3071700 1 --2- 172.21.15.29:0/30658381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f679c106f50 0x7f679c19c2b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6790011980 tx=0x7f6790011a60 comp rx=0 tx=0).stop 2024-01-24T13:30:26.816 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.555+0000 7f67a3071700 1 -- 172.21.15.29:0/30658381 shutdown_connections 2024-01-24T13:30:26.816 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.555+0000 7f67a3071700 1 --2- 172.21.15.29:0/30658381 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f678c0383a0 0x7f678c03a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:26.816 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.555+0000 7f67a3071700 1 --2- 172.21.15.29:0/30658381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f679c106f50 0x7f679c19c2b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:26.816 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.555+0000 7f67a3071700 1 -- 172.21.15.29:0/30658381 >> 172.21.15.29:0/30658381 conn(0x7f679c100980 msgr2=0x7f679c101680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:26.816 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.555+0000 7f67a3071700 1 -- 172.21.15.29:0/30658381 shutdown_connections 2024-01-24T13:30:26.816 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:24.555+0000 7f67a3071700 1 -- 172.21.15.29:0/30658381 wait complete. 2024-01-24T13:30:26.816 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for the mgr to restart... 2024-01-24T13:30:26.816 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mgr epoch 9... 2024-01-24T13:30:26.913 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:26 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:26.620+0000 7effe9b79200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-24T13:30:26.913 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:26 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:26.726+0000 7effe9b79200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-24T13:30:27.896 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:27 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:27.617+0000 7effe9b79200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-24T13:30:27.896 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:27 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:27.726+0000 7effe9b79200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-24T13:30:28.349 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:27 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:27.896+0000 7effe9b79200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-24T13:30:29.043 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:28 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:28.616+0000 7effe9b79200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-24T13:30:29.044 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:28 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:28.823+0000 7effe9b79200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-24T13:30:29.349 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:29 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:29.043+0000 7effe9b79200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-24T13:30:30.349 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:30 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:30.091+0000 7effe9b79200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-24T13:30:30.715 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:30 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:30.396+0000 7effe9b79200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-24T13:30:30.715 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:30 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:30.495+0000 7effe9b79200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-24T13:30:30.715 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:30:30 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x[40903]: 2024-01-24T13:30:30.715+0000 7effe9b79200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-24T13:30:31.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:31 smithi029 ceph-mon[40651]: Active manager daemon x restarted 2024-01-24T13:30:31.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:31 smithi029 ceph-mon[40651]: Activating manager daemon x 2024-01-24T13:30:31.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:31 smithi029 ceph-mon[40651]: osdmap e3: 0 total, 0 up, 0 in 2024-01-24T13:30:31.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:31 smithi029 ceph-mon[40651]: mgrmap e10: x(active, starting, since 0.00375308s) 2024-01-24T13:30:31.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:31 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-01-24T13:30:31.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:31 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-01-24T13:30:31.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:31 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-24T13:30:31.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:31 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-24T13:30:31.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:31 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-24T13:30:31.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:31 smithi029 ceph-mon[40651]: Manager daemon x is now available 2024-01-24T13:30:31.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:31 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:30:31.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:31 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:30:31.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:31 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-01-24T13:30:31.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:31 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-01-24T13:30:32.295 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-01-24T13:30:32.295 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-01-24T13:30:32.295 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "initialized": true 2024-01-24T13:30:32.295 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-01-24T13:30:32.295 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.174+0000 7f728b131700 1 Processor -- start 2024-01-24T13:30:32.295 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.174+0000 7f728b131700 1 -- start start 2024-01-24T13:30:32.295 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.175+0000 7f728b131700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72841064a0 0x7f72841068c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:32.295 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.175+0000 7f728b131700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7284106e90 con 0x7f72841064a0 2024-01-24T13:30:32.295 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.175+0000 7f7288ecd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72841064a0 0x7f72841068c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:32.295 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.175+0000 7f7288ecd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72841064a0 0x7f72841068c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39090/0 (socket says 172.21.15.29:39090) 2024-01-24T13:30:32.296 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.175+0000 7f7288ecd700 1 -- 172.21.15.29:0/97720371 learned_addr learned my addr 172.21.15.29:0/97720371 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:32.296 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.175+0000 7f7288ecd700 1 -- 172.21.15.29:0/97720371 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72841076a0 con 0x7f72841064a0 2024-01-24T13:30:32.296 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.175+0000 7f7288ecd700 1 --2- 172.21.15.29:0/97720371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72841064a0 0x7f72841068c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f72780098a0 tx=0x7f7278009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=355cfdcf02c304df server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:32.296 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.176+0000 7f72837fe700 1 -- 172.21.15.29:0/97720371 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7278004160 con 0x7f72841064a0 2024-01-24T13:30:32.296 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.176+0000 7f72837fe700 1 -- 172.21.15.29:0/97720371 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f72780042c0 con 0x7f72841064a0 2024-01-24T13:30:32.296 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.176+0000 7f72837fe700 1 -- 172.21.15.29:0/97720371 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f72780119b0 con 0x7f72841064a0 2024-01-24T13:30:32.296 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.177+0000 7f728b131700 1 -- 172.21.15.29:0/97720371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72841064a0 msgr2=0x7f72841068c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:32.296 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.177+0000 7f728b131700 1 --2- 172.21.15.29:0/97720371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72841064a0 0x7f72841068c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f72780098a0 tx=0x7f7278009bb0 comp rx=0 tx=0).stop 2024-01-24T13:30:32.296 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.177+0000 7f728b131700 1 -- 172.21.15.29:0/97720371 shutdown_connections 2024-01-24T13:30:32.296 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.177+0000 7f728b131700 1 --2- 172.21.15.29:0/97720371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72841064a0 0x7f72841068c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:32.296 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.177+0000 7f728b131700 1 -- 172.21.15.29:0/97720371 >> 172.21.15.29:0/97720371 conn(0x7f7284101a60 msgr2=0x7f7284103e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:32.296 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.177+0000 7f728b131700 1 -- 172.21.15.29:0/97720371 shutdown_connections 2024-01-24T13:30:32.296 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.177+0000 7f728b131700 1 -- 172.21.15.29:0/97720371 wait complete. 2024-01-24T13:30:32.296 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.177+0000 7f728b131700 1 Processor -- start 2024-01-24T13:30:32.297 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.178+0000 7f728b131700 1 -- start start 2024-01-24T13:30:32.297 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.178+0000 7f728b131700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72841064a0 0x7f728419a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:32.297 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.178+0000 7f728b131700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f728419a620 con 0x7f72841064a0 2024-01-24T13:30:32.297 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.178+0000 7f7288ecd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72841064a0 0x7f728419a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:32.297 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.178+0000 7f7288ecd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72841064a0 0x7f728419a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39104/0 (socket says 172.21.15.29:39104) 2024-01-24T13:30:32.297 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.178+0000 7f7288ecd700 1 -- 172.21.15.29:0/929272162 learned_addr learned my addr 172.21.15.29:0/929272162 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:32.297 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.178+0000 7f7288ecd700 1 -- 172.21.15.29:0/929272162 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7278009550 con 0x7f72841064a0 2024-01-24T13:30:32.297 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.178+0000 7f7288ecd700 1 --2- 172.21.15.29:0/929272162 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72841064a0 0x7f728419a0e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7278000c00 tx=0x7f7278011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:32.297 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.178+0000 7f7281ffb700 1 -- 172.21.15.29:0/929272162 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f72780039f0 con 0x7f72841064a0 2024-01-24T13:30:32.297 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.179+0000 7f728b131700 1 -- 172.21.15.29:0/929272162 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f728419a820 con 0x7f72841064a0 2024-01-24T13:30:32.297 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.179+0000 7f7281ffb700 1 -- 172.21.15.29:0/929272162 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7278003b50 con 0x7f72841064a0 2024-01-24T13:30:32.297 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.179+0000 7f7281ffb700 1 -- 172.21.15.29:0/929272162 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f727801a780 con 0x7f72841064a0 2024-01-24T13:30:32.297 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.179+0000 7f728b131700 1 -- 172.21.15.29:0/929272162 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f728419acc0 con 0x7f72841064a0 2024-01-24T13:30:32.298 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.180+0000 7f7281ffb700 1 -- 172.21.15.29:0/929272162 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 9) v1 ==== 45281+0+0 (secure 0 0 0) 0x7f727801aa40 con 0x7f72841064a0 2024-01-24T13:30:32.298 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.180+0000 7f7281ffb700 1 --2- 172.21.15.29:0/929272162 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f7274038350 0x7f727403a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:32.298 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.180+0000 7f7281ffb700 1 -- 172.21.15.29:0/929272162 --> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f727403af20 con 0x7f7274038350 2024-01-24T13:30:32.298 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.180+0000 7f7281ffb700 1 -- 172.21.15.29:0/929272162 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f727804c280 con 0x7f72841064a0 2024-01-24T13:30:32.298 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.180+0000 7f7283fff700 1 -- 172.21.15.29:0/929272162 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f7274038350 msgr2=0x7f727403a810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/1996568496 2024-01-24T13:30:32.298 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.180+0000 7f7283fff700 1 --2- 172.21.15.29:0/929272162 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f7274038350 0x7f727403a810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-24T13:30:32.298 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.381+0000 7f7283fff700 1 -- 172.21.15.29:0/929272162 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f7274038350 msgr2=0x7f727403a810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/1996568496 2024-01-24T13:30:32.298 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.381+0000 7f7283fff700 1 --2- 172.21.15.29:0/929272162 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f7274038350 0x7f727403a810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-24T13:30:32.298 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.782+0000 7f7283fff700 1 -- 172.21.15.29:0/929272162 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f7274038350 msgr2=0x7f727403a810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/1996568496 2024-01-24T13:30:32.298 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:27.782+0000 7f7283fff700 1 --2- 172.21.15.29:0/929272162 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f7274038350 0x7f727403a810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-24T13:30:32.298 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:28.583+0000 7f7283fff700 1 -- 172.21.15.29:0/929272162 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f7274038350 msgr2=0x7f727403a810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/1996568496 2024-01-24T13:30:32.298 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:28.583+0000 7f7283fff700 1 --2- 172.21.15.29:0/929272162 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f7274038350 0x7f727403a810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-24T13:30:32.298 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:30.184+0000 7f7283fff700 1 -- 172.21.15.29:0/929272162 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f7274038350 msgr2=0x7f727403a810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/1996568496 2024-01-24T13:30:32.298 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:30.184+0000 7f7283fff700 1 --2- 172.21.15.29:0/929272162 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f7274038350 0x7f727403a810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-01-24T13:30:32.298 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:30.730+0000 7f7281ffb700 1 -- 172.21.15.29:0/929272162 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mgrmap(e 10) v1 ==== 45048+0+0 (secure 0 0 0) 0x7f7278016d60 con 0x7f72841064a0 2024-01-24T13:30:32.299 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:30.730+0000 7f7281ffb700 1 -- 172.21.15.29:0/929272162 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f7274038350 msgr2=0x7f727403a810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-24T13:30:32.299 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:30.730+0000 7f7281ffb700 1 --2- 172.21.15.29:0/929272162 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f7274038350 0x7f727403a810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:32.299 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:31.733+0000 7f7281ffb700 1 -- 172.21.15.29:0/929272162 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 11) v1 ==== 45175+0+0 (secure 0 0 0) 0x7f727804dde0 con 0x7f72841064a0 2024-01-24T13:30:32.299 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:31.733+0000 7f7281ffb700 1 --2- 172.21.15.29:0/929272162 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f727403bdc0 0x7f727403e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:32.299 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:31.733+0000 7f7281ffb700 1 -- 172.21.15.29:0/929272162 --> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f727403af20 con 0x7f727403bdc0 2024-01-24T13:30:32.299 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:31.733+0000 7f7283fff700 1 --2- 172.21.15.29:0/929272162 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f727403bdc0 0x7f727403e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:32.299 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:31.734+0000 7f7283fff700 1 --2- 172.21.15.29:0/929272162 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f727403bdc0 0x7f727403e1b0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f7270003a10 tx=0x7f72700092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:32.299 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:31.734+0000 7f7281ffb700 1 -- 172.21.15.29:0/929272162 <== mgr.14152 v2:172.21.15.29:6800/2498991338 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f727403af20 con 0x7f727403bdc0 2024-01-24T13:30:32.299 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:31.742+0000 7f728b131700 1 -- 172.21.15.29:0/929272162 --> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f728419b420 con 0x7f727403bdc0 2024-01-24T13:30:32.299 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:31.742+0000 7f7281ffb700 1 -- 172.21.15.29:0/929272162 <== mgr.14152 v2:172.21.15.29:6800/2498991338 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f728419b420 con 0x7f727403bdc0 2024-01-24T13:30:32.299 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:31.742+0000 7f728b131700 1 -- 172.21.15.29:0/929272162 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f727403bdc0 msgr2=0x7f727403e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:32.299 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:31.742+0000 7f728b131700 1 --2- 172.21.15.29:0/929272162 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f727403bdc0 0x7f727403e1b0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f7270003a10 tx=0x7f72700092b0 comp rx=0 tx=0).stop 2024-01-24T13:30:32.299 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:31.743+0000 7f728b131700 1 -- 172.21.15.29:0/929272162 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72841064a0 msgr2=0x7f728419a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:32.299 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:31.743+0000 7f728b131700 1 --2- 172.21.15.29:0/929272162 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72841064a0 0x7f728419a0e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7278000c00 tx=0x7f7278011a30 comp rx=0 tx=0).stop 2024-01-24T13:30:32.299 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:31.743+0000 7f728b131700 1 -- 172.21.15.29:0/929272162 shutdown_connections 2024-01-24T13:30:32.300 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:31.743+0000 7f728b131700 1 --2- 172.21.15.29:0/929272162 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f727403bdc0 0x7f727403e1b0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:32.300 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:31.743+0000 7f728b131700 1 --2- 172.21.15.29:0/929272162 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72841064a0 0x7f728419a0e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:32.300 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:31.743+0000 7f728b131700 1 --2- 172.21.15.29:0/929272162 >> [v2:172.21.15.29:6800/1996568496,v1:172.21.15.29:6801/1996568496] conn(0x7f7274038350 0x7f727403a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:32.300 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:31.743+0000 7f728b131700 1 -- 172.21.15.29:0/929272162 >> 172.21.15.29:0/929272162 conn(0x7f7284101a60 msgr2=0x7f72841027d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:32.300 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:31.743+0000 7f728b131700 1 -- 172.21.15.29:0/929272162 shutdown_connections 2024-01-24T13:30:32.300 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:31.743+0000 7f728b131700 1 -- 172.21.15.29:0/929272162 wait complete. 2024-01-24T13:30:32.300 INFO:teuthology.orchestra.run.smithi029.stdout:mgr epoch 9 is available 2024-01-24T13:30:32.300 INFO:teuthology.orchestra.run.smithi029.stdout:Generating a dashboard self-signed certificate... 2024-01-24T13:30:33.059 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:32 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:30:33.059 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:32 smithi029 ceph-mon[40651]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-24T13:30:33.059 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:32 smithi029 ceph-mon[40651]: mgrmap e11: x(active, since 1.00624s) 2024-01-24T13:30:33.060 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:32 smithi029 ceph-mon[40651]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-24T13:30:33.569 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-01-24T13:30:33.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.675+0000 7fbb856b7700 1 Processor -- start 2024-01-24T13:30:33.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.676+0000 7fbb856b7700 1 -- start start 2024-01-24T13:30:33.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.676+0000 7fbb856b7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb801064a0 0x7fbb801068c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:33.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.676+0000 7fbb856b7700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb80106e90 con 0x7fbb801064a0 2024-01-24T13:30:33.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.676+0000 7fbb7effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb801064a0 0x7fbb801068c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:33.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.676+0000 7fbb7effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb801064a0 0x7fbb801068c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41068/0 (socket says 172.21.15.29:41068) 2024-01-24T13:30:33.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.676+0000 7fbb7effd700 1 -- 172.21.15.29:0/2380042193 learned_addr learned my addr 172.21.15.29:0/2380042193 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:33.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.677+0000 7fbb7effd700 1 -- 172.21.15.29:0/2380042193 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb801076a0 con 0x7fbb801064a0 2024-01-24T13:30:33.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.677+0000 7fbb7effd700 1 --2- 172.21.15.29:0/2380042193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb801064a0 0x7fbb801068c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbb700098a0 tx=0x7fbb70009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=758f145d32c97c5c server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:33.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.677+0000 7fbb7dffb700 1 -- 172.21.15.29:0/2380042193 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbb70004160 con 0x7fbb801064a0 2024-01-24T13:30:33.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.677+0000 7fbb7dffb700 1 -- 172.21.15.29:0/2380042193 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbb700042c0 con 0x7fbb801064a0 2024-01-24T13:30:33.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.677+0000 7fbb7dffb700 1 -- 172.21.15.29:0/2380042193 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbb700119b0 con 0x7fbb801064a0 2024-01-24T13:30:33.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.678+0000 7fbb856b7700 1 -- 172.21.15.29:0/2380042193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb801064a0 msgr2=0x7fbb801068c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:33.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.678+0000 7fbb856b7700 1 --2- 172.21.15.29:0/2380042193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb801064a0 0x7fbb801068c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbb700098a0 tx=0x7fbb70009bb0 comp rx=0 tx=0).stop 2024-01-24T13:30:33.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.678+0000 7fbb856b7700 1 -- 172.21.15.29:0/2380042193 shutdown_connections 2024-01-24T13:30:33.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.678+0000 7fbb856b7700 1 --2- 172.21.15.29:0/2380042193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb801064a0 0x7fbb801068c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:33.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.678+0000 7fbb856b7700 1 -- 172.21.15.29:0/2380042193 >> 172.21.15.29:0/2380042193 conn(0x7fbb80101a60 msgr2=0x7fbb80103e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:33.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.678+0000 7fbb856b7700 1 -- 172.21.15.29:0/2380042193 shutdown_connections 2024-01-24T13:30:33.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.678+0000 7fbb856b7700 1 -- 172.21.15.29:0/2380042193 wait complete. 2024-01-24T13:30:33.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.679+0000 7fbb856b7700 1 Processor -- start 2024-01-24T13:30:33.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.679+0000 7fbb856b7700 1 -- start start 2024-01-24T13:30:33.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.680+0000 7fbb856b7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb801064a0 0x7fbb8019a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:33.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.680+0000 7fbb856b7700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb8019a660 con 0x7fbb801064a0 2024-01-24T13:30:33.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.680+0000 7fbb7effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb801064a0 0x7fbb8019a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:33.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.680+0000 7fbb7effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb801064a0 0x7fbb8019a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41072/0 (socket says 172.21.15.29:41072) 2024-01-24T13:30:33.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.680+0000 7fbb7effd700 1 -- 172.21.15.29:0/4273177427 learned_addr learned my addr 172.21.15.29:0/4273177427 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:33.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.680+0000 7fbb7effd700 1 -- 172.21.15.29:0/4273177427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb70009550 con 0x7fbb801064a0 2024-01-24T13:30:33.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.680+0000 7fbb7effd700 1 --2- 172.21.15.29:0/4273177427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb801064a0 0x7fbb8019a120 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbb70000c00 tx=0x7fbb70011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:33.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.681+0000 7fbb67fff700 1 -- 172.21.15.29:0/4273177427 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbb700039f0 con 0x7fbb801064a0 2024-01-24T13:30:33.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.681+0000 7fbb67fff700 1 -- 172.21.15.29:0/4273177427 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbb70003b50 con 0x7fbb801064a0 2024-01-24T13:30:33.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.681+0000 7fbb67fff700 1 -- 172.21.15.29:0/4273177427 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbb7001a780 con 0x7fbb801064a0 2024-01-24T13:30:33.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.681+0000 7fbb856b7700 1 -- 172.21.15.29:0/4273177427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb8019a860 con 0x7fbb801064a0 2024-01-24T13:30:33.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.681+0000 7fbb856b7700 1 -- 172.21.15.29:0/4273177427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb8019ad00 con 0x7fbb801064a0 2024-01-24T13:30:33.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.682+0000 7fbb67fff700 1 -- 172.21.15.29:0/4273177427 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 11) v1 ==== 45175+0+0 (secure 0 0 0) 0x7fbb7001aa40 con 0x7fbb801064a0 2024-01-24T13:30:33.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.682+0000 7fbb856b7700 1 -- 172.21.15.29:0/4273177427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb8010a8d0 con 0x7fbb801064a0 2024-01-24T13:30:33.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.682+0000 7fbb67fff700 1 --2- 172.21.15.29:0/4273177427 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fbb68038220 0x7fbb6803a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:33.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.682+0000 7fbb67fff700 1 -- 172.21.15.29:0/4273177427 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fbb7004c120 con 0x7fbb801064a0 2024-01-24T13:30:33.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.682+0000 7fbb7e7fc700 1 --2- 172.21.15.29:0/4273177427 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fbb68038220 0x7fbb6803a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:33.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.683+0000 7fbb7e7fc700 1 --2- 172.21.15.29:0/4273177427 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fbb68038220 0x7fbb6803a6e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbb74006fd0 tx=0x7fbb74006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:33.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.687+0000 7fbb67fff700 1 -- 172.21.15.29:0/4273177427 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fbb700237b0 con 0x7fbb801064a0 2024-01-24T13:30:33.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:32.912+0000 7fbb856b7700 1 -- 172.21.15.29:0/4273177427 --> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb8019dae0 con 0x7fbb68038220 2024-01-24T13:30:33.573 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:33.028+0000 7fbb67fff700 1 -- 172.21.15.29:0/4273177427 <== mgr.14152 v2:172.21.15.29:6800/2498991338 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fbb8019dae0 con 0x7fbb68038220 2024-01-24T13:30:33.573 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:33.032+0000 7fbb856b7700 1 -- 172.21.15.29:0/4273177427 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fbb68038220 msgr2=0x7fbb6803a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:33.573 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:33.032+0000 7fbb856b7700 1 --2- 172.21.15.29:0/4273177427 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fbb68038220 0x7fbb6803a6e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbb74006fd0 tx=0x7fbb74006e40 comp rx=0 tx=0).stop 2024-01-24T13:30:33.574 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:33.032+0000 7fbb856b7700 1 -- 172.21.15.29:0/4273177427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb801064a0 msgr2=0x7fbb8019a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:33.574 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:33.032+0000 7fbb856b7700 1 --2- 172.21.15.29:0/4273177427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb801064a0 0x7fbb8019a120 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbb70000c00 tx=0x7fbb70011a30 comp rx=0 tx=0).stop 2024-01-24T13:30:33.574 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:33.032+0000 7fbb856b7700 1 -- 172.21.15.29:0/4273177427 shutdown_connections 2024-01-24T13:30:33.574 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:33.032+0000 7fbb856b7700 1 --2- 172.21.15.29:0/4273177427 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fbb68038220 0x7fbb6803a6e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:33.574 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:33.032+0000 7fbb856b7700 1 --2- 172.21.15.29:0/4273177427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb801064a0 0x7fbb8019a120 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:33.574 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:33.032+0000 7fbb856b7700 1 -- 172.21.15.29:0/4273177427 >> 172.21.15.29:0/4273177427 conn(0x7fbb80101a60 msgr2=0x7fbb801027f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:33.574 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:33.032+0000 7fbb856b7700 1 -- 172.21.15.29:0/4273177427 shutdown_connections 2024-01-24T13:30:33.574 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:33.032+0000 7fbb856b7700 1 -- 172.21.15.29:0/4273177427 wait complete. 2024-01-24T13:30:33.574 INFO:teuthology.orchestra.run.smithi029.stdout:Creating initial admin user... 2024-01-24T13:30:34.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:33 smithi029 ceph-mon[40651]: [24/Jan/2024:13:30:32] ENGINE Bus STARTING 2024-01-24T13:30:34.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:33 smithi029 ceph-mon[40651]: [24/Jan/2024:13:30:32] ENGINE Serving on https://172.21.15.29:7150 2024-01-24T13:30:34.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:33 smithi029 ceph-mon[40651]: [24/Jan/2024:13:30:32] ENGINE Serving on http://172.21.15.29:8765 2024-01-24T13:30:34.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:33 smithi029 ceph-mon[40651]: [24/Jan/2024:13:30:32] ENGINE Bus STARTED 2024-01-24T13:30:34.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:33 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:30:34.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:33 smithi029 ceph-mon[40651]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:30:34.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:33 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:30:34.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:33 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:30:34.866 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:34 smithi029 ceph-mon[40651]: mgrmap e12: x(active, since 3s) 2024-01-24T13:30:34.866 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:34 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:30:34.957 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$LVC4k06vE1BX8WxR577U/eIX1lsT53jRXrj.VuqvjtCXWcDIeGX1O", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1706103034, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-01-24T13:30:34.994 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.101+0000 7f2721f31700 1 Processor -- start 2024-01-24T13:30:34.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.101+0000 7f2721f31700 1 -- start start 2024-01-24T13:30:34.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.101+0000 7f2721f31700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f271c07bec0 0x7f271c07a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:34.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.101+0000 7f2721f31700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f271c07a8f0 con 0x7f271c07bec0 2024-01-24T13:30:34.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.101+0000 7f271b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f271c07bec0 0x7f271c07a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:34.995 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.101+0000 7f271b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f271c07bec0 0x7f271c07a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41074/0 (socket says 172.21.15.29:41074) 2024-01-24T13:30:34.996 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.101+0000 7f271b7fe700 1 -- 172.21.15.29:0/3551954702 learned_addr learned my addr 172.21.15.29:0/3551954702 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:34.996 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.102+0000 7f271b7fe700 1 -- 172.21.15.29:0/3551954702 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f271c07aa30 con 0x7f271c07bec0 2024-01-24T13:30:34.996 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.102+0000 7f271b7fe700 1 --2- 172.21.15.29:0/3551954702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f271c07bec0 0x7f271c07a320 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f270c0098a0 tx=0x7f270c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3036bb04f1865333 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:34.996 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.102+0000 7f271a7fc700 1 -- 172.21.15.29:0/3551954702 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f270c004160 con 0x7f271c07bec0 2024-01-24T13:30:34.996 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.102+0000 7f271a7fc700 1 -- 172.21.15.29:0/3551954702 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f270c0042c0 con 0x7f271c07bec0 2024-01-24T13:30:34.996 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.102+0000 7f271a7fc700 1 -- 172.21.15.29:0/3551954702 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f270c0119b0 con 0x7f271c07bec0 2024-01-24T13:30:34.997 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.103+0000 7f2721f31700 1 -- 172.21.15.29:0/3551954702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f271c07bec0 msgr2=0x7f271c07a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:34.997 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.103+0000 7f2721f31700 1 --2- 172.21.15.29:0/3551954702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f271c07bec0 0x7f271c07a320 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f270c0098a0 tx=0x7f270c009bb0 comp rx=0 tx=0).stop 2024-01-24T13:30:34.997 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.103+0000 7f2721f31700 1 -- 172.21.15.29:0/3551954702 shutdown_connections 2024-01-24T13:30:34.997 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.103+0000 7f2721f31700 1 --2- 172.21.15.29:0/3551954702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f271c07bec0 0x7f271c07a320 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:34.997 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.103+0000 7f2721f31700 1 -- 172.21.15.29:0/3551954702 >> 172.21.15.29:0/3551954702 conn(0x7f271c101a60 msgr2=0x7f271c103e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:34.997 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.103+0000 7f2721f31700 1 -- 172.21.15.29:0/3551954702 shutdown_connections 2024-01-24T13:30:34.997 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.103+0000 7f2721f31700 1 -- 172.21.15.29:0/3551954702 wait complete. 2024-01-24T13:30:34.997 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.103+0000 7f2721f31700 1 Processor -- start 2024-01-24T13:30:34.997 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.104+0000 7f2721f31700 1 -- start start 2024-01-24T13:30:34.997 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.104+0000 7f2721f31700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f271c07bec0 0x7f271c1a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:34.998 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.104+0000 7f2721f31700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f271c1a0c90 con 0x7f271c07bec0 2024-01-24T13:30:34.998 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.104+0000 7f271b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f271c07bec0 0x7f271c1a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:34.998 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.104+0000 7f271b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f271c07bec0 0x7f271c1a0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41090/0 (socket says 172.21.15.29:41090) 2024-01-24T13:30:34.998 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.104+0000 7f271b7fe700 1 -- 172.21.15.29:0/2108894774 learned_addr learned my addr 172.21.15.29:0/2108894774 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:34.998 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.104+0000 7f271b7fe700 1 -- 172.21.15.29:0/2108894774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f270c009550 con 0x7f271c07bec0 2024-01-24T13:30:34.998 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.104+0000 7f271b7fe700 1 --2- 172.21.15.29:0/2108894774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f271c07bec0 0x7f271c1a0750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f270c000c00 tx=0x7f270c011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:34.998 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.104+0000 7f2718ff9700 1 -- 172.21.15.29:0/2108894774 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f270c0039f0 con 0x7f271c07bec0 2024-01-24T13:30:34.998 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.105+0000 7f2718ff9700 1 -- 172.21.15.29:0/2108894774 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f270c003b50 con 0x7f271c07bec0 2024-01-24T13:30:34.999 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.105+0000 7f2721f31700 1 -- 172.21.15.29:0/2108894774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f271c1a0e90 con 0x7f271c07bec0 2024-01-24T13:30:34.999 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.105+0000 7f2718ff9700 1 -- 172.21.15.29:0/2108894774 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f270c01a780 con 0x7f271c07bec0 2024-01-24T13:30:34.999 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.105+0000 7f2721f31700 1 -- 172.21.15.29:0/2108894774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f271c1a1330 con 0x7f271c07bec0 2024-01-24T13:30:34.999 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.106+0000 7f2718ff9700 1 -- 172.21.15.29:0/2108894774 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 45281+0+0 (secure 0 0 0) 0x7f270c01a9f0 con 0x7f271c07bec0 2024-01-24T13:30:34.999 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.106+0000 7f2721f31700 1 -- 172.21.15.29:0/2108894774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f271c108470 con 0x7f271c07bec0 2024-01-24T13:30:34.999 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.106+0000 7f2718ff9700 1 --2- 172.21.15.29:0/2108894774 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f27080383a0 0x7f270803a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:34.999 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.106+0000 7f2718ff9700 1 -- 172.21.15.29:0/2108894774 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f270c04c440 con 0x7f271c07bec0 2024-01-24T13:30:34.999 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.106+0000 7f271affd700 1 --2- 172.21.15.29:0/2108894774 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f27080383a0 0x7f270803a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:35.000 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.106+0000 7f271affd700 1 --2- 172.21.15.29:0/2108894774 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f27080383a0 0x7f270803a860 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2704006fd0 tx=0x7f2704006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:35.000 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.111+0000 7f2718ff9700 1 -- 172.21.15.29:0/2108894774 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f270c0237b0 con 0x7f271c07bec0 2024-01-24T13:30:35.000 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.334+0000 7f2721f31700 1 -- 172.21.15.29:0/2108894774 --> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f271c062190 con 0x7f27080383a0 2024-01-24T13:30:35.000 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.588+0000 7f2718ff9700 1 -- 172.21.15.29:0/2108894774 <== mgr.14152 v2:172.21.15.29:6800/2498991338 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f271c062190 con 0x7f27080383a0 2024-01-24T13:30:35.000 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.590+0000 7f2721f31700 1 -- 172.21.15.29:0/2108894774 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f27080383a0 msgr2=0x7f270803a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:35.000 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.590+0000 7f2721f31700 1 --2- 172.21.15.29:0/2108894774 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f27080383a0 0x7f270803a860 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2704006fd0 tx=0x7f2704006e40 comp rx=0 tx=0).stop 2024-01-24T13:30:35.000 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.590+0000 7f2721f31700 1 -- 172.21.15.29:0/2108894774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f271c07bec0 msgr2=0x7f271c1a0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:35.000 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.590+0000 7f2721f31700 1 --2- 172.21.15.29:0/2108894774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f271c07bec0 0x7f271c1a0750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f270c000c00 tx=0x7f270c011a30 comp rx=0 tx=0).stop 2024-01-24T13:30:35.000 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.590+0000 7f2721f31700 1 -- 172.21.15.29:0/2108894774 shutdown_connections 2024-01-24T13:30:35.000 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.590+0000 7f2721f31700 1 --2- 172.21.15.29:0/2108894774 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f27080383a0 0x7f270803a860 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:35.000 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.590+0000 7f2721f31700 1 --2- 172.21.15.29:0/2108894774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f271c07bec0 0x7f271c1a0750 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:35.000 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.590+0000 7f2721f31700 1 -- 172.21.15.29:0/2108894774 >> 172.21.15.29:0/2108894774 conn(0x7f271c101a60 msgr2=0x7f271c107660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:35.001 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.590+0000 7f2721f31700 1 -- 172.21.15.29:0/2108894774 shutdown_connections 2024-01-24T13:30:35.001 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:34.590+0000 7f2721f31700 1 -- 172.21.15.29:0/2108894774 wait complete. 2024-01-24T13:30:35.001 INFO:teuthology.orchestra.run.smithi029.stdout:Fetching dashboard port number... 2024-01-24T13:30:35.901 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 8443 2024-01-24T13:30:35.901 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.330+0000 7fb761873700 1 Processor -- start 2024-01-24T13:30:35.902 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.330+0000 7fb761873700 1 -- start start 2024-01-24T13:30:35.902 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.331+0000 7fb761873700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb75c106f30 0x7fb75c109320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:35.902 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.331+0000 7fb761873700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75c079e50 con 0x7fb75c106f30 2024-01-24T13:30:35.902 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.331+0000 7fb75affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb75c106f30 0x7fb75c109320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:35.902 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.331+0000 7fb75affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb75c106f30 0x7fb75c109320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41102/0 (socket says 172.21.15.29:41102) 2024-01-24T13:30:35.902 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.331+0000 7fb75affd700 1 -- 172.21.15.29:0/4223715908 learned_addr learned my addr 172.21.15.29:0/4223715908 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:35.902 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.331+0000 7fb75affd700 1 -- 172.21.15.29:0/4223715908 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb75c109860 con 0x7fb75c106f30 2024-01-24T13:30:35.902 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.331+0000 7fb75affd700 1 --2- 172.21.15.29:0/4223715908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb75c106f30 0x7fb75c109320 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb74c0098a0 tx=0x7fb74c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd2706d0a5c8c139 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:35.902 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.332+0000 7fb75a7fc700 1 -- 172.21.15.29:0/4223715908 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb74c004160 con 0x7fb75c106f30 2024-01-24T13:30:35.902 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.332+0000 7fb75a7fc700 1 -- 172.21.15.29:0/4223715908 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb74c0042c0 con 0x7fb75c106f30 2024-01-24T13:30:35.902 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.332+0000 7fb75a7fc700 1 -- 172.21.15.29:0/4223715908 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb74c0119b0 con 0x7fb75c106f30 2024-01-24T13:30:35.903 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.332+0000 7fb761873700 1 -- 172.21.15.29:0/4223715908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb75c106f30 msgr2=0x7fb75c109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:35.903 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.332+0000 7fb761873700 1 --2- 172.21.15.29:0/4223715908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb75c106f30 0x7fb75c109320 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb74c0098a0 tx=0x7fb74c009bb0 comp rx=0 tx=0).stop 2024-01-24T13:30:35.903 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.332+0000 7fb761873700 1 -- 172.21.15.29:0/4223715908 shutdown_connections 2024-01-24T13:30:35.903 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.332+0000 7fb761873700 1 --2- 172.21.15.29:0/4223715908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb75c106f30 0x7fb75c109320 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:35.903 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.332+0000 7fb761873700 1 -- 172.21.15.29:0/4223715908 >> 172.21.15.29:0/4223715908 conn(0x7fb75c100940 msgr2=0x7fb75c102d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:35.903 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.332+0000 7fb761873700 1 -- 172.21.15.29:0/4223715908 shutdown_connections 2024-01-24T13:30:35.903 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.332+0000 7fb761873700 1 -- 172.21.15.29:0/4223715908 wait complete. 2024-01-24T13:30:35.903 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.333+0000 7fb761873700 1 Processor -- start 2024-01-24T13:30:35.903 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.333+0000 7fb761873700 1 -- start start 2024-01-24T13:30:35.903 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.333+0000 7fb761873700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb75c106f30 0x7fb75c106890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:35.903 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.333+0000 7fb761873700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75c106dd0 con 0x7fb75c106f30 2024-01-24T13:30:35.903 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.333+0000 7fb75affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb75c106f30 0x7fb75c106890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:35.903 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.333+0000 7fb75affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb75c106f30 0x7fb75c106890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41118/0 (socket says 172.21.15.29:41118) 2024-01-24T13:30:35.904 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.334+0000 7fb75affd700 1 -- 172.21.15.29:0/3081514849 learned_addr learned my addr 172.21.15.29:0/3081514849 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:35.904 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.334+0000 7fb75affd700 1 -- 172.21.15.29:0/3081514849 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb74c009550 con 0x7fb75c106f30 2024-01-24T13:30:35.904 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.334+0000 7fb75affd700 1 --2- 172.21.15.29:0/3081514849 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb75c106f30 0x7fb75c106890 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb74c006af0 tx=0x7fb74c011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:35.904 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.334+0000 7fb760871700 1 -- 172.21.15.29:0/3081514849 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb74c003a40 con 0x7fb75c106f30 2024-01-24T13:30:35.904 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.334+0000 7fb760871700 1 -- 172.21.15.29:0/3081514849 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb74c003ba0 con 0x7fb75c106f30 2024-01-24T13:30:35.904 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.334+0000 7fb761873700 1 -- 172.21.15.29:0/3081514849 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb75c104ee0 con 0x7fb75c106f30 2024-01-24T13:30:35.904 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.334+0000 7fb760871700 1 -- 172.21.15.29:0/3081514849 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb74c0196c0 con 0x7fb75c106f30 2024-01-24T13:30:35.904 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.334+0000 7fb761873700 1 -- 172.21.15.29:0/3081514849 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb75c105380 con 0x7fb75c106f30 2024-01-24T13:30:35.904 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.335+0000 7fb760871700 1 -- 172.21.15.29:0/3081514849 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 45281+0+0 (secure 0 0 0) 0x7fb74c019980 con 0x7fb75c106f30 2024-01-24T13:30:35.904 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.335+0000 7fb761873700 1 -- 172.21.15.29:0/3081514849 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb75c10a640 con 0x7fb75c106f30 2024-01-24T13:30:35.904 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.335+0000 7fb760871700 1 --2- 172.21.15.29:0/3081514849 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb748038300 0x7fb74803a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:35.904 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.335+0000 7fb760871700 1 -- 172.21.15.29:0/3081514849 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb74c04c750 con 0x7fb75c106f30 2024-01-24T13:30:35.904 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.335+0000 7fb7527fc700 1 --2- 172.21.15.29:0/3081514849 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb748038300 0x7fb74803a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:35.904 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.336+0000 7fb7527fc700 1 --2- 172.21.15.29:0/3081514849 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb748038300 0x7fb74803a7c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb744006fd0 tx=0x7fb744006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:35.905 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.341+0000 7fb760871700 1 -- 172.21.15.29:0/3081514849 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fb74c018760 con 0x7fb75c106f30 2024-01-24T13:30:35.905 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.546+0000 7fb761873700 1 -- 172.21.15.29:0/3081514849 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fb75c10a860 con 0x7fb75c106f30 2024-01-24T13:30:35.905 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.547+0000 7fb760871700 1 -- 172.21.15.29:0/3081514849 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7fb74c018350 con 0x7fb75c106f30 2024-01-24T13:30:35.905 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.549+0000 7fb761873700 1 -- 172.21.15.29:0/3081514849 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb748038300 msgr2=0x7fb74803a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:35.905 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.549+0000 7fb761873700 1 --2- 172.21.15.29:0/3081514849 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb748038300 0x7fb74803a7c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb744006fd0 tx=0x7fb744006e40 comp rx=0 tx=0).stop 2024-01-24T13:30:35.905 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.549+0000 7fb761873700 1 -- 172.21.15.29:0/3081514849 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb75c106f30 msgr2=0x7fb75c106890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:35.905 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.549+0000 7fb761873700 1 --2- 172.21.15.29:0/3081514849 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb75c106f30 0x7fb75c106890 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb74c006af0 tx=0x7fb74c011a10 comp rx=0 tx=0).stop 2024-01-24T13:30:35.905 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.549+0000 7fb761873700 1 -- 172.21.15.29:0/3081514849 shutdown_connections 2024-01-24T13:30:35.905 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.549+0000 7fb761873700 1 --2- 172.21.15.29:0/3081514849 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb748038300 0x7fb74803a7c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:35.905 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.549+0000 7fb761873700 1 --2- 172.21.15.29:0/3081514849 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb75c106f30 0x7fb75c106890 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:35.905 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.549+0000 7fb761873700 1 -- 172.21.15.29:0/3081514849 >> 172.21.15.29:0/3081514849 conn(0x7fb75c100940 msgr2=0x7fb75c1016e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:35.905 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.549+0000 7fb761873700 1 -- 172.21.15.29:0/3081514849 shutdown_connections 2024-01-24T13:30:35.905 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:35.549+0000 7fb761873700 1 -- 172.21.15.29:0/3081514849 wait complete. 2024-01-24T13:30:35.912 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-01-24T13:30:35.913 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout disabled 2024-01-24T13:30:35.922 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-01-24T13:30:35.922 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-01-24T13:30:35.922 INFO:teuthology.orchestra.run.smithi029.stdout:firewalld.service is not enabled 2024-01-24T13:30:35.922 INFO:teuthology.orchestra.run.smithi029.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-01-24T13:30:35.924 INFO:teuthology.orchestra.run.smithi029.stdout:Ceph Dashboard is now available at: 2024-01-24T13:30:35.924 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:30:35.924 INFO:teuthology.orchestra.run.smithi029.stdout: URL: https://smithi029:8443/ 2024-01-24T13:30:35.924 INFO:teuthology.orchestra.run.smithi029.stdout: User: admin 2024-01-24T13:30:35.924 INFO:teuthology.orchestra.run.smithi029.stdout: Password: f091tehksg 2024-01-24T13:30:35.924 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:30:35.925 INFO:teuthology.orchestra.run.smithi029.stdout:Saving cluster configuration to /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/config directory 2024-01-24T13:30:35.925 INFO:teuthology.orchestra.run.smithi029.stdout:Enabling autotune for osd_memory_target 2024-01-24T13:30:36.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:36 smithi029 ceph-mon[40651]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:30:36.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:36 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/3081514849' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-01-24T13:30:36.868 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.304+0000 7f3d818ff700 1 Processor -- start 2024-01-24T13:30:36.868 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.304+0000 7f3d818ff700 1 -- start start 2024-01-24T13:30:36.869 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.305+0000 7f3d818ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d7c106f50 0x7f3d7c109340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:36.869 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.305+0000 7f3d818ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7c079e30 con 0x7f3d7c106f50 2024-01-24T13:30:36.869 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.305+0000 7f3d7affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d7c106f50 0x7f3d7c109340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:36.869 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.305+0000 7f3d7affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d7c106f50 0x7f3d7c109340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41124/0 (socket says 172.21.15.29:41124) 2024-01-24T13:30:36.869 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.305+0000 7f3d7affd700 1 -- 172.21.15.29:0/3650586358 learned_addr learned my addr 172.21.15.29:0/3650586358 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:36.869 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.305+0000 7f3d7affd700 1 -- 172.21.15.29:0/3650586358 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d7c109880 con 0x7f3d7c106f50 2024-01-24T13:30:36.869 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.305+0000 7f3d7affd700 1 --2- 172.21.15.29:0/3650586358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d7c106f50 0x7f3d7c109340 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c0098a0 tx=0x7f3d6c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=130d4337cb3b6b62 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:36.869 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.305+0000 7f3d7a7fc700 1 -- 172.21.15.29:0/3650586358 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3d6c004160 con 0x7f3d7c106f50 2024-01-24T13:30:36.869 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.306+0000 7f3d7a7fc700 1 -- 172.21.15.29:0/3650586358 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d6c0042c0 con 0x7f3d7c106f50 2024-01-24T13:30:36.869 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.306+0000 7f3d7a7fc700 1 -- 172.21.15.29:0/3650586358 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3d6c0119b0 con 0x7f3d7c106f50 2024-01-24T13:30:36.869 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.306+0000 7f3d818ff700 1 -- 172.21.15.29:0/3650586358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d7c106f50 msgr2=0x7f3d7c109340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:36.869 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.306+0000 7f3d818ff700 1 --2- 172.21.15.29:0/3650586358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d7c106f50 0x7f3d7c109340 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c0098a0 tx=0x7f3d6c009bb0 comp rx=0 tx=0).stop 2024-01-24T13:30:36.869 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.306+0000 7f3d818ff700 1 -- 172.21.15.29:0/3650586358 shutdown_connections 2024-01-24T13:30:36.870 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.306+0000 7f3d818ff700 1 --2- 172.21.15.29:0/3650586358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d7c106f50 0x7f3d7c109340 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:36.870 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.306+0000 7f3d818ff700 1 -- 172.21.15.29:0/3650586358 >> 172.21.15.29:0/3650586358 conn(0x7f3d7c100940 msgr2=0x7f3d7c102d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:36.870 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.306+0000 7f3d818ff700 1 -- 172.21.15.29:0/3650586358 shutdown_connections 2024-01-24T13:30:36.870 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.306+0000 7f3d818ff700 1 -- 172.21.15.29:0/3650586358 wait complete. 2024-01-24T13:30:36.870 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.307+0000 7f3d818ff700 1 Processor -- start 2024-01-24T13:30:36.870 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.307+0000 7f3d818ff700 1 -- start start 2024-01-24T13:30:36.870 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.307+0000 7f3d818ff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d7c106f50 0x7f3d7c1068b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:36.870 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.307+0000 7f3d818ff700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7c106df0 con 0x7f3d7c106f50 2024-01-24T13:30:36.870 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.308+0000 7f3d7affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d7c106f50 0x7f3d7c1068b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:36.870 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.308+0000 7f3d7affd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d7c106f50 0x7f3d7c1068b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41126/0 (socket says 172.21.15.29:41126) 2024-01-24T13:30:36.870 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.308+0000 7f3d7affd700 1 -- 172.21.15.29:0/1058720893 learned_addr learned my addr 172.21.15.29:0/1058720893 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:36.870 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.308+0000 7f3d7affd700 1 -- 172.21.15.29:0/1058720893 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d6c009550 con 0x7f3d7c106f50 2024-01-24T13:30:36.870 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.308+0000 7f3d7affd700 1 --2- 172.21.15.29:0/1058720893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d7c106f50 0x7f3d7c1068b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c006af0 tx=0x7f3d6c011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:36.871 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.308+0000 7f3d808fd700 1 -- 172.21.15.29:0/1058720893 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3d6c003a40 con 0x7f3d7c106f50 2024-01-24T13:30:36.871 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.308+0000 7f3d808fd700 1 -- 172.21.15.29:0/1058720893 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d6c003ba0 con 0x7f3d7c106f50 2024-01-24T13:30:36.871 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.308+0000 7f3d808fd700 1 -- 172.21.15.29:0/1058720893 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3d6c0196c0 con 0x7f3d7c106f50 2024-01-24T13:30:36.871 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.308+0000 7f3d818ff700 1 -- 172.21.15.29:0/1058720893 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d7c104f00 con 0x7f3d7c106f50 2024-01-24T13:30:36.871 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.308+0000 7f3d818ff700 1 -- 172.21.15.29:0/1058720893 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d7c1053a0 con 0x7f3d7c106f50 2024-01-24T13:30:36.871 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.309+0000 7f3d808fd700 1 -- 172.21.15.29:0/1058720893 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) v1 ==== 45281+0+0 (secure 0 0 0) 0x7f3d6c019930 con 0x7f3d7c106f50 2024-01-24T13:30:36.871 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.309+0000 7f3d818ff700 1 -- 172.21.15.29:0/1058720893 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d7c10a660 con 0x7f3d7c106f50 2024-01-24T13:30:36.871 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.310+0000 7f3d808fd700 1 --2- 172.21.15.29:0/1058720893 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f3d68038350 0x7f3d6803a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:36.871 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.310+0000 7f3d808fd700 1 -- 172.21.15.29:0/1058720893 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3d6c04c750 con 0x7f3d7c106f50 2024-01-24T13:30:36.871 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.310+0000 7f3d727fc700 1 --2- 172.21.15.29:0/1058720893 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f3d68038350 0x7f3d6803a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:36.871 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.310+0000 7f3d727fc700 1 --2- 172.21.15.29:0/1058720893 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f3d68038350 0x7f3d6803a810 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3d64006fd0 tx=0x7f3d64006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:36.871 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.315+0000 7f3d808fd700 1 -- 172.21.15.29:0/1058720893 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f3d6c023b40 con 0x7f3d7c106f50 2024-01-24T13:30:36.871 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.520+0000 7f3d818ff700 1 -- 172.21.15.29:0/1058720893 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f3d7c10a880 con 0x7f3d7c106f50 2024-01-24T13:30:36.871 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.520+0000 7f3d808fd700 1 -- 172.21.15.29:0/1058720893 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f3d6c023b40 con 0x7f3d7c106f50 2024-01-24T13:30:36.872 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.522+0000 7f3d818ff700 1 -- 172.21.15.29:0/1058720893 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f3d68038350 msgr2=0x7f3d6803a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:36.872 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.522+0000 7f3d818ff700 1 --2- 172.21.15.29:0/1058720893 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f3d68038350 0x7f3d6803a810 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3d64006fd0 tx=0x7f3d64006e40 comp rx=0 tx=0).stop 2024-01-24T13:30:36.872 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.522+0000 7f3d818ff700 1 -- 172.21.15.29:0/1058720893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d7c106f50 msgr2=0x7f3d7c1068b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:36.872 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.522+0000 7f3d818ff700 1 --2- 172.21.15.29:0/1058720893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d7c106f50 0x7f3d7c1068b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c006af0 tx=0x7f3d6c011a10 comp rx=0 tx=0).stop 2024-01-24T13:30:36.872 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.522+0000 7f3d818ff700 1 -- 172.21.15.29:0/1058720893 shutdown_connections 2024-01-24T13:30:36.872 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.522+0000 7f3d818ff700 1 --2- 172.21.15.29:0/1058720893 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f3d68038350 0x7f3d6803a810 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:36.872 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.522+0000 7f3d818ff700 1 --2- 172.21.15.29:0/1058720893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d7c106f50 0x7f3d7c1068b0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:36.872 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.522+0000 7f3d818ff700 1 -- 172.21.15.29:0/1058720893 >> 172.21.15.29:0/1058720893 conn(0x7f3d7c100940 msgr2=0x7f3d7c1016e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:36.872 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.522+0000 7f3d818ff700 1 -- 172.21.15.29:0/1058720893 shutdown_connections 2024-01-24T13:30:36.872 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:36.522+0000 7f3d818ff700 1 -- 172.21.15.29:0/1058720893 wait complete. 2024-01-24T13:30:37.911 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.231+0000 7f3ce080a700 1 Processor -- start 2024-01-24T13:30:37.911 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.232+0000 7f3ce080a700 1 -- start start 2024-01-24T13:30:37.911 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.232+0000 7f3ce080a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3cdc106410 0x7f3cdc106830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:37.911 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.232+0000 7f3ce080a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cdc106e00 con 0x7f3cdc106410 2024-01-24T13:30:37.911 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.232+0000 7f3cda59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3cdc106410 0x7f3cdc106830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:37.911 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.232+0000 7f3cda59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3cdc106410 0x7f3cdc106830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41140/0 (socket says 172.21.15.29:41140) 2024-01-24T13:30:37.911 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.232+0000 7f3cda59c700 1 -- 172.21.15.29:0/1104091552 learned_addr learned my addr 172.21.15.29:0/1104091552 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:37.911 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.232+0000 7f3cda59c700 1 -- 172.21.15.29:0/1104091552 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cdc107610 con 0x7f3cdc106410 2024-01-24T13:30:37.911 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.232+0000 7f3cda59c700 1 --2- 172.21.15.29:0/1104091552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3cdc106410 0x7f3cdc106830 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc0098a0 tx=0x7f3ccc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a202582407d5e25 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:37.912 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.233+0000 7f3cd9d9b700 1 -- 172.21.15.29:0/1104091552 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3ccc004160 con 0x7f3cdc106410 2024-01-24T13:30:37.912 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.233+0000 7f3cd9d9b700 1 -- 172.21.15.29:0/1104091552 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3ccc0042c0 con 0x7f3cdc106410 2024-01-24T13:30:37.912 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.233+0000 7f3cd9d9b700 1 -- 172.21.15.29:0/1104091552 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3ccc0119b0 con 0x7f3cdc106410 2024-01-24T13:30:37.912 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.233+0000 7f3ce080a700 1 -- 172.21.15.29:0/1104091552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3cdc106410 msgr2=0x7f3cdc106830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:37.912 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.233+0000 7f3ce080a700 1 --2- 172.21.15.29:0/1104091552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3cdc106410 0x7f3cdc106830 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc0098a0 tx=0x7f3ccc009bb0 comp rx=0 tx=0).stop 2024-01-24T13:30:37.912 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.233+0000 7f3ce080a700 1 -- 172.21.15.29:0/1104091552 shutdown_connections 2024-01-24T13:30:37.912 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.233+0000 7f3ce080a700 1 --2- 172.21.15.29:0/1104091552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3cdc106410 0x7f3cdc106830 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:37.912 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.233+0000 7f3ce080a700 1 -- 172.21.15.29:0/1104091552 >> 172.21.15.29:0/1104091552 conn(0x7f3cdc1019d0 msgr2=0x7f3cdc103df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:37.912 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.233+0000 7f3ce080a700 1 -- 172.21.15.29:0/1104091552 shutdown_connections 2024-01-24T13:30:37.912 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.233+0000 7f3ce080a700 1 -- 172.21.15.29:0/1104091552 wait complete. 2024-01-24T13:30:37.912 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.234+0000 7f3ce080a700 1 Processor -- start 2024-01-24T13:30:37.912 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.234+0000 7f3ce080a700 1 -- start start 2024-01-24T13:30:37.913 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.234+0000 7f3ce080a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3cdc106410 0x7f3cdc07bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:37.913 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.234+0000 7f3ce080a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cdc07c120 con 0x7f3cdc106410 2024-01-24T13:30:37.913 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.235+0000 7f3cda59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3cdc106410 0x7f3cdc07bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:37.913 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.235+0000 7f3cda59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3cdc106410 0x7f3cdc07bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41146/0 (socket says 172.21.15.29:41146) 2024-01-24T13:30:37.913 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.235+0000 7f3cda59c700 1 -- 172.21.15.29:0/174647079 learned_addr learned my addr 172.21.15.29:0/174647079 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:37.913 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.235+0000 7f3cda59c700 1 -- 172.21.15.29:0/174647079 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ccc009550 con 0x7f3cdc106410 2024-01-24T13:30:37.913 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.235+0000 7f3cda59c700 1 --2- 172.21.15.29:0/174647079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3cdc106410 0x7f3cdc07bbe0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc003e60 tx=0x7f3ccc011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:37.913 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.235+0000 7f3cd37fe700 1 -- 172.21.15.29:0/174647079 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3ccc003970 con 0x7f3cdc106410 2024-01-24T13:30:37.913 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.235+0000 7f3cd37fe700 1 -- 172.21.15.29:0/174647079 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3ccc003ad0 con 0x7f3cdc106410 2024-01-24T13:30:37.913 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.235+0000 7f3cd37fe700 1 -- 172.21.15.29:0/174647079 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3ccc0195e0 con 0x7f3cdc106410 2024-01-24T13:30:37.913 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.235+0000 7f3ce080a700 1 -- 172.21.15.29:0/174647079 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cdc07a280 con 0x7f3cdc106410 2024-01-24T13:30:37.913 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.235+0000 7f3ce080a700 1 -- 172.21.15.29:0/174647079 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cdc07a780 con 0x7f3cdc106410 2024-01-24T13:30:37.913 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.236+0000 7f3cd37fe700 1 -- 172.21.15.29:0/174647079 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45324+0+0 (secure 0 0 0) 0x7f3ccc01a430 con 0x7f3cdc106410 2024-01-24T13:30:37.914 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.236+0000 7f3ce080a700 1 -- 172.21.15.29:0/174647079 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cdc10a840 con 0x7f3cdc106410 2024-01-24T13:30:37.914 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.237+0000 7f3cd37fe700 1 --2- 172.21.15.29:0/174647079 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f3cc80383f0 0x7f3cc803a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:37.914 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.237+0000 7f3cd37fe700 1 -- 172.21.15.29:0/174647079 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3ccc04c4c0 con 0x7f3cdc106410 2024-01-24T13:30:37.914 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.237+0000 7f3cd3fff700 1 --2- 172.21.15.29:0/174647079 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f3cc80383f0 0x7f3cc803a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:37.914 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.237+0000 7f3cd3fff700 1 --2- 172.21.15.29:0/174647079 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f3cc80383f0 0x7f3cc803a8b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4006fd0 tx=0x7f3cc4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:37.914 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.242+0000 7f3cd37fe700 1 -- 172.21.15.29:0/174647079 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f3ccc003c40 con 0x7f3cdc106410 2024-01-24T13:30:37.914 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.561+0000 7f3ce080a700 1 -- 172.21.15.29:0/174647079 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f3cdc07b090 con 0x7f3cdc106410 2024-01-24T13:30:37.914 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.564+0000 7f3cd37fe700 1 -- 172.21.15.29:0/174647079 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7f3ccc0212a0 con 0x7f3cdc106410 2024-01-24T13:30:37.914 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.566+0000 7f3ce080a700 1 -- 172.21.15.29:0/174647079 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f3cc80383f0 msgr2=0x7f3cc803a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:37.914 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.566+0000 7f3ce080a700 1 --2- 172.21.15.29:0/174647079 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f3cc80383f0 0x7f3cc803a8b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4006fd0 tx=0x7f3cc4006e40 comp rx=0 tx=0).stop 2024-01-24T13:30:37.914 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.566+0000 7f3ce080a700 1 -- 172.21.15.29:0/174647079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3cdc106410 msgr2=0x7f3cdc07bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:37.914 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.566+0000 7f3ce080a700 1 --2- 172.21.15.29:0/174647079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3cdc106410 0x7f3cdc07bbe0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc003e60 tx=0x7f3ccc011aa0 comp rx=0 tx=0).stop 2024-01-24T13:30:37.914 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.566+0000 7f3ce080a700 1 -- 172.21.15.29:0/174647079 shutdown_connections 2024-01-24T13:30:37.914 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.566+0000 7f3ce080a700 1 --2- 172.21.15.29:0/174647079 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f3cc80383f0 0x7f3cc803a8b0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:37.915 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.566+0000 7f3ce080a700 1 --2- 172.21.15.29:0/174647079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3cdc106410 0x7f3cdc07bbe0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:37.915 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.566+0000 7f3ce080a700 1 -- 172.21.15.29:0/174647079 >> 172.21.15.29:0/174647079 conn(0x7f3cdc1019d0 msgr2=0x7f3cdc103070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:37.915 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.566+0000 7f3ce080a700 1 -- 172.21.15.29:0/174647079 shutdown_connections 2024-01-24T13:30:37.915 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-01-24T13:30:37.566+0000 7f3ce080a700 1 -- 172.21.15.29:0/174647079 wait complete. 2024-01-24T13:30:37.915 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-01-24T13:30:37.915 INFO:teuthology.orchestra.run.smithi029.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-01-24T13:30:37.915 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:30:37.915 INFO:teuthology.orchestra.run.smithi029.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-01-24T13:30:37.915 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:30:37.915 INFO:teuthology.orchestra.run.smithi029.stdout:Or, if you are only running a single cluster on this host: 2024-01-24T13:30:37.915 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:30:37.915 INFO:teuthology.orchestra.run.smithi029.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-01-24T13:30:37.915 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:30:37.916 INFO:teuthology.orchestra.run.smithi029.stdout:Please consider enabling telemetry to help improve Ceph: 2024-01-24T13:30:37.916 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:30:37.916 INFO:teuthology.orchestra.run.smithi029.stdout: ceph telemetry on 2024-01-24T13:30:37.916 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:30:37.916 INFO:teuthology.orchestra.run.smithi029.stdout:For more information see: 2024-01-24T13:30:37.916 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:30:37.916 INFO:teuthology.orchestra.run.smithi029.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-01-24T13:30:37.916 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:30:37.916 INFO:teuthology.orchestra.run.smithi029.stdout:Bootstrap complete. 2024-01-24T13:30:37.956 INFO:tasks.cephadm:Fetching config... 2024-01-24T13:30:37.956 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-24T13:30:37.957 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-01-24T13:30:37.978 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-01-24T13:30:37.978 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-24T13:30:37.978 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-01-24T13:30:38.039 INFO:tasks.cephadm:Fetching mon keyring... 2024-01-24T13:30:38.039 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-24T13:30:38.039 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.a/keyring of=/dev/stdout 2024-01-24T13:30:38.113 INFO:tasks.cephadm:Fetching pub ssh key... 2024-01-24T13:30:38.113 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-24T13:30:38.113 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-01-24T13:30:38.175 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-01-24T13:30:38.175 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDqf55o8JDj6jP24erpNFQ8fci7GwghB0vocUucS9zXqshz89guKmvquPG2e0mtkDHjc3tYKITRAJDxxw6Y9oXwrOHx1qncYX70Et3dy/L7mLW8SkeHUXShfOEc4IN9okr6YNcMOsWyP32VdyyhZUsUNSUcQi6OqzXGWkFfCAOBM8sffqFhxmQ4NnknjMClbjWKmzb7QzNPVu3qoySGAKa/DgOLvtfGfbogJfKqp8yTn98FqnJNShUck64pJ7wHb9FlPy/jUV62GUziCDddc5/h/63G2OjznUyoA8x967lz76PQOf/tDWuoLyK6QpJsccsRwyRljzJGaUm8Ge0TmnOykkqRBKwTbAtobjWTlEz+gQS1CIE6jBldfyXHU5+mFXDs2h5LohsorNiMv/lNRKNhoPT2ulJRs6axKHQ7IvJcpk/seAhNwkej5Iqf6i88IUQgCGuDt7jhJGqA5Gk+Vm5qi7KUnyZNhxKSr6LHUPYpPV8g3PNyox+e97BbclQAvrE= ceph-83afcf5e-babc-11ee-95b1-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-24T13:30:38.244 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:38 smithi029 ceph-mon[40651]: mgrmap e13: x(active, since 6s) 2024-01-24T13:30:38.244 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:38 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/174647079' entity='client.admin' 2024-01-24T13:30:38.261 INFO:teuthology.orchestra.run.smithi029.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDqf55o8JDj6jP24erpNFQ8fci7GwghB0vocUucS9zXqshz89guKmvquPG2e0mtkDHjc3tYKITRAJDxxw6Y9oXwrOHx1qncYX70Et3dy/L7mLW8SkeHUXShfOEc4IN9okr6YNcMOsWyP32VdyyhZUsUNSUcQi6OqzXGWkFfCAOBM8sffqFhxmQ4NnknjMClbjWKmzb7QzNPVu3qoySGAKa/DgOLvtfGfbogJfKqp8yTn98FqnJNShUck64pJ7wHb9FlPy/jUV62GUziCDddc5/h/63G2OjznUyoA8x967lz76PQOf/tDWuoLyK6QpJsccsRwyRljzJGaUm8Ge0TmnOykkqRBKwTbAtobjWTlEz+gQS1CIE6jBldfyXHU5+mFXDs2h5LohsorNiMv/lNRKNhoPT2ulJRs6axKHQ7IvJcpk/seAhNwkej5Iqf6i88IUQgCGuDt7jhJGqA5Gk+Vm5qi7KUnyZNhxKSr6LHUPYpPV8g3PNyox+e97BbclQAvrE= ceph-83afcf5e-babc-11ee-95b1-87774f69a715 2024-01-24T13:30:38.275 DEBUG:teuthology.orchestra.run.smithi163:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDqf55o8JDj6jP24erpNFQ8fci7GwghB0vocUucS9zXqshz89guKmvquPG2e0mtkDHjc3tYKITRAJDxxw6Y9oXwrOHx1qncYX70Et3dy/L7mLW8SkeHUXShfOEc4IN9okr6YNcMOsWyP32VdyyhZUsUNSUcQi6OqzXGWkFfCAOBM8sffqFhxmQ4NnknjMClbjWKmzb7QzNPVu3qoySGAKa/DgOLvtfGfbogJfKqp8yTn98FqnJNShUck64pJ7wHb9FlPy/jUV62GUziCDddc5/h/63G2OjznUyoA8x967lz76PQOf/tDWuoLyK6QpJsccsRwyRljzJGaUm8Ge0TmnOykkqRBKwTbAtobjWTlEz+gQS1CIE6jBldfyXHU5+mFXDs2h5LohsorNiMv/lNRKNhoPT2ulJRs6axKHQ7IvJcpk/seAhNwkej5Iqf6i88IUQgCGuDt7jhJGqA5Gk+Vm5qi7KUnyZNhxKSr6LHUPYpPV8g3PNyox+e97BbclQAvrE= ceph-83afcf5e-babc-11ee-95b1-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-24T13:30:38.323 INFO:teuthology.orchestra.run.smithi163.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDqf55o8JDj6jP24erpNFQ8fci7GwghB0vocUucS9zXqshz89guKmvquPG2e0mtkDHjc3tYKITRAJDxxw6Y9oXwrOHx1qncYX70Et3dy/L7mLW8SkeHUXShfOEc4IN9okr6YNcMOsWyP32VdyyhZUsUNSUcQi6OqzXGWkFfCAOBM8sffqFhxmQ4NnknjMClbjWKmzb7QzNPVu3qoySGAKa/DgOLvtfGfbogJfKqp8yTn98FqnJNShUck64pJ7wHb9FlPy/jUV62GUziCDddc5/h/63G2OjznUyoA8x967lz76PQOf/tDWuoLyK6QpJsccsRwyRljzJGaUm8Ge0TmnOykkqRBKwTbAtobjWTlEz+gQS1CIE6jBldfyXHU5+mFXDs2h5LohsorNiMv/lNRKNhoPT2ulJRs6axKHQ7IvJcpk/seAhNwkej5Iqf6i88IUQgCGuDt7jhJGqA5Gk+Vm5qi7KUnyZNhxKSr6LHUPYpPV8g3PNyox+e97BbclQAvrE= ceph-83afcf5e-babc-11ee-95b1-87774f69a715 2024-01-24T13:30:38.336 DEBUG:teuthology.orchestra.run.smithi187:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDqf55o8JDj6jP24erpNFQ8fci7GwghB0vocUucS9zXqshz89guKmvquPG2e0mtkDHjc3tYKITRAJDxxw6Y9oXwrOHx1qncYX70Et3dy/L7mLW8SkeHUXShfOEc4IN9okr6YNcMOsWyP32VdyyhZUsUNSUcQi6OqzXGWkFfCAOBM8sffqFhxmQ4NnknjMClbjWKmzb7QzNPVu3qoySGAKa/DgOLvtfGfbogJfKqp8yTn98FqnJNShUck64pJ7wHb9FlPy/jUV62GUziCDddc5/h/63G2OjznUyoA8x967lz76PQOf/tDWuoLyK6QpJsccsRwyRljzJGaUm8Ge0TmnOykkqRBKwTbAtobjWTlEz+gQS1CIE6jBldfyXHU5+mFXDs2h5LohsorNiMv/lNRKNhoPT2ulJRs6axKHQ7IvJcpk/seAhNwkej5Iqf6i88IUQgCGuDt7jhJGqA5Gk+Vm5qi7KUnyZNhxKSr6LHUPYpPV8g3PNyox+e97BbclQAvrE= ceph-83afcf5e-babc-11ee-95b1-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-24T13:30:38.382 INFO:teuthology.orchestra.run.smithi187.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDqf55o8JDj6jP24erpNFQ8fci7GwghB0vocUucS9zXqshz89guKmvquPG2e0mtkDHjc3tYKITRAJDxxw6Y9oXwrOHx1qncYX70Et3dy/L7mLW8SkeHUXShfOEc4IN9okr6YNcMOsWyP32VdyyhZUsUNSUcQi6OqzXGWkFfCAOBM8sffqFhxmQ4NnknjMClbjWKmzb7QzNPVu3qoySGAKa/DgOLvtfGfbogJfKqp8yTn98FqnJNShUck64pJ7wHb9FlPy/jUV62GUziCDddc5/h/63G2OjznUyoA8x967lz76PQOf/tDWuoLyK6QpJsccsRwyRljzJGaUm8Ge0TmnOykkqRBKwTbAtobjWTlEz+gQS1CIE6jBldfyXHU5+mFXDs2h5LohsorNiMv/lNRKNhoPT2ulJRs6axKHQ7IvJcpk/seAhNwkej5Iqf6i88IUQgCGuDt7jhJGqA5Gk+Vm5qi7KUnyZNhxKSr6LHUPYpPV8g3PNyox+e97BbclQAvrE= ceph-83afcf5e-babc-11ee-95b1-87774f69a715 2024-01-24T13:30:38.395 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-01-24T13:30:38.649 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T13:30:39.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:39.879+0000 7f4dae58d700 1 -- 172.21.15.29:0/2632338893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da8101320 msgr2=0x7f4da8103710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:39.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:39.879+0000 7f4dae58d700 1 --2- 172.21.15.29:0/2632338893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da8101320 0x7f4da8103710 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4d98009910 tx=0x7f4d98009c20 comp rx=0 tx=0).stop 2024-01-24T13:30:39.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:39.879+0000 7f4dae58d700 1 -- 172.21.15.29:0/2632338893 shutdown_connections 2024-01-24T13:30:39.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:39.879+0000 7f4dae58d700 1 --2- 172.21.15.29:0/2632338893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da8101320 0x7f4da8103710 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:39.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:39.879+0000 7f4dae58d700 1 -- 172.21.15.29:0/2632338893 >> 172.21.15.29:0/2632338893 conn(0x7f4da80facd0 msgr2=0x7f4da80fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:39.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:39.879+0000 7f4dae58d700 1 -- 172.21.15.29:0/2632338893 shutdown_connections 2024-01-24T13:30:39.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:39.879+0000 7f4dae58d700 1 -- 172.21.15.29:0/2632338893 wait complete. 2024-01-24T13:30:39.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:39.880+0000 7f4dae58d700 1 Processor -- start 2024-01-24T13:30:39.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:39.880+0000 7f4dae58d700 1 -- start start 2024-01-24T13:30:39.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:39.881+0000 7f4dae58d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da8101320 0x7f4da8100c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:39.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:39.881+0000 7f4dae58d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da81011c0 con 0x7f4da8101320 2024-01-24T13:30:39.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:39.881+0000 7f4da7fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da8101320 0x7f4da8100c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:39.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:39.881+0000 7f4da7fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da8101320 0x7f4da8100c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41160/0 (socket says 172.21.15.29:41160) 2024-01-24T13:30:39.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:39.881+0000 7f4da7fff700 1 -- 172.21.15.29:0/1751764423 learned_addr learned my addr 172.21.15.29:0/1751764423 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:39.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:39.881+0000 7f4da7fff700 1 -- 172.21.15.29:0/1751764423 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d980095f0 con 0x7f4da8101320 2024-01-24T13:30:39.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:39.881+0000 7f4da7fff700 1 --2- 172.21.15.29:0/1751764423 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da8101320 0x7f4da8100c80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4d98005f50 tx=0x7f4d980057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:39.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:39.881+0000 7f4da57fa700 1 -- 172.21.15.29:0/1751764423 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d9801d070 con 0x7f4da8101320 2024-01-24T13:30:39.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:39.882+0000 7f4da57fa700 1 -- 172.21.15.29:0/1751764423 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4d98022470 con 0x7f4da8101320 2024-01-24T13:30:39.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:39.882+0000 7f4dae58d700 1 -- 172.21.15.29:0/1751764423 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4da80ff2d0 con 0x7f4da8101320 2024-01-24T13:30:39.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:39.882+0000 7f4da57fa700 1 -- 172.21.15.29:0/1751764423 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d9800f5f0 con 0x7f4da8101320 2024-01-24T13:30:39.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:39.882+0000 7f4dae58d700 1 -- 172.21.15.29:0/1751764423 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4da80ff770 con 0x7f4da8101320 2024-01-24T13:30:39.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:39.883+0000 7f4dae58d700 1 -- 172.21.15.29:0/1751764423 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4da8062370 con 0x7f4da8101320 2024-01-24T13:30:39.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:39.883+0000 7f4da57fa700 1 -- 172.21.15.29:0/1751764423 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45324+0+0 (secure 0 0 0) 0x7f4d9800f750 con 0x7f4da8101320 2024-01-24T13:30:39.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:39.883+0000 7f4da57fa700 1 --2- 172.21.15.29:0/1751764423 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f4d900383f0 0x7f4d9003a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:39.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:39.883+0000 7f4da57fa700 1 -- 172.21.15.29:0/1751764423 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4d9804c4b0 con 0x7f4da8101320 2024-01-24T13:30:39.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:39.883+0000 7f4da77fe700 1 --2- 172.21.15.29:0/1751764423 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f4d900383f0 0x7f4d9003a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:39.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:39.884+0000 7f4da77fe700 1 --2- 172.21.15.29:0/1751764423 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f4d900383f0 0x7f4d9003a8b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c006fd0 tx=0x7f4d9c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:39.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:39.889+0000 7f4da57fa700 1 -- 172.21.15.29:0/1751764423 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f4d9802a360 con 0x7f4da8101320 2024-01-24T13:30:40.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:40.106+0000 7f4dae58d700 1 -- 172.21.15.29:0/1751764423 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f4da818f0a0 con 0x7f4da8101320 2024-01-24T13:30:40.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:40.110+0000 7f4da57fa700 1 -- 172.21.15.29:0/1751764423 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f4d98027030 con 0x7f4da8101320 2024-01-24T13:30:40.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:40.112+0000 7f4dae58d700 1 -- 172.21.15.29:0/1751764423 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f4d900383f0 msgr2=0x7f4d9003a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:40.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:40.112+0000 7f4dae58d700 1 --2- 172.21.15.29:0/1751764423 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f4d900383f0 0x7f4d9003a8b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c006fd0 tx=0x7f4d9c006e40 comp rx=0 tx=0).stop 2024-01-24T13:30:40.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:40.112+0000 7f4dae58d700 1 -- 172.21.15.29:0/1751764423 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da8101320 msgr2=0x7f4da8100c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:40.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:40.112+0000 7f4dae58d700 1 --2- 172.21.15.29:0/1751764423 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da8101320 0x7f4da8100c80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4d98005f50 tx=0x7f4d980057c0 comp rx=0 tx=0).stop 2024-01-24T13:30:40.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:40.112+0000 7f4dae58d700 1 -- 172.21.15.29:0/1751764423 shutdown_connections 2024-01-24T13:30:40.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:40.112+0000 7f4dae58d700 1 --2- 172.21.15.29:0/1751764423 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f4d900383f0 0x7f4d9003a8b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:40.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:40.112+0000 7f4dae58d700 1 --2- 172.21.15.29:0/1751764423 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da8101320 0x7f4da8100c80 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:40.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:40.112+0000 7f4dae58d700 1 -- 172.21.15.29:0/1751764423 >> 172.21.15.29:0/1751764423 conn(0x7f4da80facd0 msgr2=0x7f4da80fba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:40.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:40.113+0000 7f4dae58d700 1 -- 172.21.15.29:0/1751764423 shutdown_connections 2024-01-24T13:30:40.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:40.113+0000 7f4dae58d700 1 -- 172.21.15.29:0/1751764423 wait complete. 2024-01-24T13:30:40.498 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-01-24T13:30:40.498 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-01-24T13:30:40.771 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T13:30:41.062 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:40 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:30:41.062 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:40 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:30:41.062 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:40 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi029", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:30:41.062 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:40 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:30:41.062 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:40 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:30:41.062 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:40 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:30:41.062 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:40 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/1751764423' entity='client.admin' 2024-01-24T13:30:41.063 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:40 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:30:41.063 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:40 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:30:41.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:41.822+0000 7f7eecbe8700 1 -- 172.21.15.29:0/3227550541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ee8101300 msgr2=0x7f7ee81036f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:41.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:41.823+0000 7f7eecbe8700 1 --2- 172.21.15.29:0/3227550541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ee8101300 0x7f7ee81036f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8009910 tx=0x7f7ed8009c20 comp rx=0 tx=0).stop 2024-01-24T13:30:41.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:41.823+0000 7f7eecbe8700 1 -- 172.21.15.29:0/3227550541 shutdown_connections 2024-01-24T13:30:41.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:41.823+0000 7f7eecbe8700 1 --2- 172.21.15.29:0/3227550541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ee8101300 0x7f7ee81036f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:41.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:41.823+0000 7f7eecbe8700 1 -- 172.21.15.29:0/3227550541 >> 172.21.15.29:0/3227550541 conn(0x7f7ee80facb0 msgr2=0x7f7ee80fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:41.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:41.823+0000 7f7eecbe8700 1 -- 172.21.15.29:0/3227550541 shutdown_connections 2024-01-24T13:30:41.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:41.823+0000 7f7eecbe8700 1 -- 172.21.15.29:0/3227550541 wait complete. 2024-01-24T13:30:41.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:41.823+0000 7f7eecbe8700 1 Processor -- start 2024-01-24T13:30:41.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:41.824+0000 7f7eecbe8700 1 -- start start 2024-01-24T13:30:41.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:41.824+0000 7f7eecbe8700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ee8101300 0x7f7ee8100c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:41.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:41.824+0000 7f7eecbe8700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee81011a0 con 0x7f7ee8101300 2024-01-24T13:30:41.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:41.824+0000 7f7ee659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ee8101300 0x7f7ee8100c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:41.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:41.824+0000 7f7ee659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ee8101300 0x7f7ee8100c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46994/0 (socket says 172.21.15.29:46994) 2024-01-24T13:30:41.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:41.824+0000 7f7ee659c700 1 -- 172.21.15.29:0/737389296 learned_addr learned my addr 172.21.15.29:0/737389296 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:41.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:41.824+0000 7f7ee659c700 1 -- 172.21.15.29:0/737389296 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed80095f0 con 0x7f7ee8101300 2024-01-24T13:30:41.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:41.825+0000 7f7ee659c700 1 --2- 172.21.15.29:0/737389296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ee8101300 0x7f7ee8100c60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8004f40 tx=0x7f7ed80057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:41.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:41.825+0000 7f7edffff700 1 -- 172.21.15.29:0/737389296 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ed801d070 con 0x7f7ee8101300 2024-01-24T13:30:41.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:41.825+0000 7f7edffff700 1 -- 172.21.15.29:0/737389296 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ed80044e0 con 0x7f7ee8101300 2024-01-24T13:30:41.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:41.825+0000 7f7edffff700 1 -- 172.21.15.29:0/737389296 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ed8003a20 con 0x7f7ee8101300 2024-01-24T13:30:41.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:41.825+0000 7f7eecbe8700 1 -- 172.21.15.29:0/737389296 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ee80ff2b0 con 0x7f7ee8101300 2024-01-24T13:30:41.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:41.825+0000 7f7eecbe8700 1 -- 172.21.15.29:0/737389296 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ee80ff6f0 con 0x7f7ee8101300 2024-01-24T13:30:41.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:41.826+0000 7f7edffff700 1 -- 172.21.15.29:0/737389296 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45324+0+0 (secure 0 0 0) 0x7f7ed8003c50 con 0x7f7ee8101300 2024-01-24T13:30:41.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:41.826+0000 7f7edffff700 1 --2- 172.21.15.29:0/737389296 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f7ed40383f0 0x7f7ed403a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:41.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:41.826+0000 7f7edffff700 1 -- 172.21.15.29:0/737389296 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7ed800b960 con 0x7f7ee8101300 2024-01-24T13:30:41.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:41.826+0000 7f7eddd9b700 1 --2- 172.21.15.29:0/737389296 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f7ed40383f0 0x7f7ed403a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:41.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:41.826+0000 7f7edd59a700 1 -- 172.21.15.29:0/737389296 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ecc0052f0 con 0x7f7ee8101300 2024-01-24T13:30:41.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:41.827+0000 7f7eddd9b700 1 --2- 172.21.15.29:0/737389296 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f7ed40383f0 0x7f7ed403a8b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7ed0006fd0 tx=0x7f7ed0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:41.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:41.831+0000 7f7edffff700 1 -- 172.21.15.29:0/737389296 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f7ed800f460 con 0x7f7ee8101300 2024-01-24T13:30:42.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:42.047+0000 7f7edd59a700 1 -- 172.21.15.29:0/737389296 --> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ecc000bc0 con 0x7f7ed40383f0 2024-01-24T13:30:42.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:42.050+0000 7f7edffff700 1 -- 172.21.15.29:0/737389296 <== mgr.14152 v2:172.21.15.29:6800/2498991338 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7ecc000bc0 con 0x7f7ed40383f0 2024-01-24T13:30:42.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:42.052+0000 7f7edd59a700 1 -- 172.21.15.29:0/737389296 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f7ed40383f0 msgr2=0x7f7ed403a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:42.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:42.052+0000 7f7edd59a700 1 --2- 172.21.15.29:0/737389296 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f7ed40383f0 0x7f7ed403a8b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7ed0006fd0 tx=0x7f7ed0006e40 comp rx=0 tx=0).stop 2024-01-24T13:30:42.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:42.052+0000 7f7edd59a700 1 -- 172.21.15.29:0/737389296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ee8101300 msgr2=0x7f7ee8100c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:42.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:42.052+0000 7f7edd59a700 1 --2- 172.21.15.29:0/737389296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ee8101300 0x7f7ee8100c60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8004f40 tx=0x7f7ed80057c0 comp rx=0 tx=0).stop 2024-01-24T13:30:42.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:42.053+0000 7f7edd59a700 1 -- 172.21.15.29:0/737389296 shutdown_connections 2024-01-24T13:30:42.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:42.053+0000 7f7edd59a700 1 --2- 172.21.15.29:0/737389296 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f7ed40383f0 0x7f7ed403a8b0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:42.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:42.053+0000 7f7edd59a700 1 --2- 172.21.15.29:0/737389296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ee8101300 0x7f7ee8100c60 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:42.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:42.053+0000 7f7edd59a700 1 -- 172.21.15.29:0/737389296 >> 172.21.15.29:0/737389296 conn(0x7f7ee80facb0 msgr2=0x7f7ee80fba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:42.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:42.053+0000 7f7edd59a700 1 -- 172.21.15.29:0/737389296 shutdown_connections 2024-01-24T13:30:42.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:42.053+0000 7f7edd59a700 1 -- 172.21.15.29:0/737389296 wait complete. 2024-01-24T13:30:42.474 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi163 2024-01-24T13:30:42.474 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-01-24T13:30:42.474 DEBUG:teuthology.orchestra.run.smithi163:> dd of=/etc/ceph/ceph.conf 2024-01-24T13:30:42.495 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-01-24T13:30:42.495 DEBUG:teuthology.orchestra.run.smithi163:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-24T13:30:42.555 INFO:tasks.cephadm:Adding host smithi163 to orchestrator... 2024-01-24T13:30:42.555 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph orch host add smithi163 2024-01-24T13:30:42.825 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T13:30:43.319 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:43 smithi029 ceph-mon[40651]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:30:43.319 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:43 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:30:43.319 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:43 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:30:43.320 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:43 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:30:43.320 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:43 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:30:43.320 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:43 smithi029 ceph-mon[40651]: Updating smithi029:/etc/ceph/ceph.conf 2024-01-24T13:30:43.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:43.835+0000 7f48357af700 1 -- 172.21.15.29:0/1481411619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48301067d0 msgr2=0x7f4830106bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:43.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:43.835+0000 7f48357af700 1 --2- 172.21.15.29:0/1481411619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48301067d0 0x7f4830106bb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4820009910 tx=0x7f4820009c20 comp rx=0 tx=0).stop 2024-01-24T13:30:43.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:43.835+0000 7f48357af700 1 -- 172.21.15.29:0/1481411619 shutdown_connections 2024-01-24T13:30:43.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:43.835+0000 7f48357af700 1 --2- 172.21.15.29:0/1481411619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48301067d0 0x7f4830106bb0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:43.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:43.835+0000 7f48357af700 1 -- 172.21.15.29:0/1481411619 >> 172.21.15.29:0/1481411619 conn(0x7f48300fc300 msgr2=0x7f48300fe720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:43.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:43.835+0000 7f48357af700 1 -- 172.21.15.29:0/1481411619 shutdown_connections 2024-01-24T13:30:43.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:43.835+0000 7f48357af700 1 -- 172.21.15.29:0/1481411619 wait complete. 2024-01-24T13:30:43.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:43.836+0000 7f48357af700 1 Processor -- start 2024-01-24T13:30:43.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:43.836+0000 7f48357af700 1 -- start start 2024-01-24T13:30:43.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:43.836+0000 7f48357af700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48301067d0 0x7f4830195ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:43.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:43.836+0000 7f48357af700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48301961e0 con 0x7f48301067d0 2024-01-24T13:30:43.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:43.837+0000 7f482effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48301067d0 0x7f4830195ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:43.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:43.837+0000 7f482effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48301067d0 0x7f4830195ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47004/0 (socket says 172.21.15.29:47004) 2024-01-24T13:30:43.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:43.837+0000 7f482effd700 1 -- 172.21.15.29:0/2982261128 learned_addr learned my addr 172.21.15.29:0/2982261128 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:43.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:43.837+0000 7f482effd700 1 -- 172.21.15.29:0/2982261128 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48200095f0 con 0x7f48301067d0 2024-01-24T13:30:43.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:43.837+0000 7f482effd700 1 --2- 172.21.15.29:0/2982261128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48301067d0 0x7f4830195ca0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4820004f40 tx=0x7f4820005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:43.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:43.837+0000 7f4827fff700 1 -- 172.21.15.29:0/2982261128 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f482001d070 con 0x7f48301067d0 2024-01-24T13:30:43.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:43.837+0000 7f4827fff700 1 -- 172.21.15.29:0/2982261128 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4820004370 con 0x7f48301067d0 2024-01-24T13:30:43.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:43.837+0000 7f48357af700 1 -- 172.21.15.29:0/2982261128 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4830196440 con 0x7f48301067d0 2024-01-24T13:30:43.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:43.837+0000 7f4827fff700 1 -- 172.21.15.29:0/2982261128 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4820003970 con 0x7f48301067d0 2024-01-24T13:30:43.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:43.837+0000 7f48357af700 1 -- 172.21.15.29:0/2982261128 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48301968c0 con 0x7f48301067d0 2024-01-24T13:30:43.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:43.838+0000 7f4827fff700 1 -- 172.21.15.29:0/2982261128 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45324+0+0 (secure 0 0 0) 0x7f4820003c30 con 0x7f48301067d0 2024-01-24T13:30:43.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:43.838+0000 7f4827fff700 1 --2- 172.21.15.29:0/2982261128 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f481c0383a0 0x7f481c03a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:43.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:43.838+0000 7f4827fff700 1 -- 172.21.15.29:0/2982261128 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f482000bcd0 con 0x7f48301067d0 2024-01-24T13:30:43.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:43.839+0000 7f482e7fc700 1 --2- 172.21.15.29:0/2982261128 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f481c0383a0 0x7f481c03a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:43.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:43.839+0000 7f48357af700 1 -- 172.21.15.29:0/2982261128 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4830062370 con 0x7f48301067d0 2024-01-24T13:30:43.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:43.839+0000 7f482e7fc700 1 --2- 172.21.15.29:0/2982261128 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f481c0383a0 0x7f481c03a860 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4818006fd0 tx=0x7f4818006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:43.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:43.844+0000 7f4827fff700 1 -- 172.21.15.29:0/2982261128 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f4820026070 con 0x7f48301067d0 2024-01-24T13:30:44.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:44.054+0000 7f48357af700 1 -- 172.21.15.29:0/2982261128 --> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi163", "target": ["mon-mgr", ""]}) v1 -- 0x7f48301995a0 con 0x7f481c0383a0 2024-01-24T13:30:44.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:44 smithi029 ceph-mon[40651]: Updating smithi029:/var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-24T13:30:44.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:44 smithi029 ceph-mon[40651]: Updating smithi029:/etc/ceph/ceph.client.admin.keyring 2024-01-24T13:30:44.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:44 smithi029 ceph-mon[40651]: Updating smithi029:/var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/config/ceph.client.admin.keyring 2024-01-24T13:30:44.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:44 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:30:44.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:44 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:30:44.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:44 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:30:45.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:45 smithi029 ceph-mon[40651]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi163", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:30:46.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:46.366+0000 7f4827fff700 1 -- 172.21.15.29:0/2982261128 <== mgr.14152 v2:172.21.15.29:6800/2498991338 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f48301995a0 con 0x7f481c0383a0 2024-01-24T13:30:46.368 INFO:teuthology.orchestra.run.smithi029.stdout:Added host 'smithi163' with addr '172.21.15.163' 2024-01-24T13:30:46.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:46.369+0000 7f48357af700 1 -- 172.21.15.29:0/2982261128 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f481c0383a0 msgr2=0x7f481c03a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:46.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:46.369+0000 7f48357af700 1 --2- 172.21.15.29:0/2982261128 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f481c0383a0 0x7f481c03a860 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4818006fd0 tx=0x7f4818006e40 comp rx=0 tx=0).stop 2024-01-24T13:30:46.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:46.369+0000 7f48357af700 1 -- 172.21.15.29:0/2982261128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48301067d0 msgr2=0x7f4830195ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:46.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:46.369+0000 7f48357af700 1 --2- 172.21.15.29:0/2982261128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48301067d0 0x7f4830195ca0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4820004f40 tx=0x7f4820005020 comp rx=0 tx=0).stop 2024-01-24T13:30:46.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:46.369+0000 7f48357af700 1 -- 172.21.15.29:0/2982261128 shutdown_connections 2024-01-24T13:30:46.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:46.369+0000 7f48357af700 1 --2- 172.21.15.29:0/2982261128 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f481c0383a0 0x7f481c03a860 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:46.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:46.369+0000 7f48357af700 1 --2- 172.21.15.29:0/2982261128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48301067d0 0x7f4830195ca0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:46.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:46.369+0000 7f48357af700 1 -- 172.21.15.29:0/2982261128 >> 172.21.15.29:0/2982261128 conn(0x7f48300fc300 msgr2=0x7f4830108c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:46.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:46.370+0000 7f48357af700 1 -- 172.21.15.29:0/2982261128 shutdown_connections 2024-01-24T13:30:46.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:46.370+0000 7f48357af700 1 -- 172.21.15.29:0/2982261128 wait complete. 2024-01-24T13:30:46.395 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:46 smithi029 ceph-mon[40651]: Deploying cephadm binary to smithi163 2024-01-24T13:30:46.757 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph orch host ls --format=json 2024-01-24T13:30:47.008 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T13:30:47.435 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:47 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:30:47.435 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:47 smithi029 ceph-mon[40651]: Added host smithi163 2024-01-24T13:30:47.436 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:47 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:30:47.436 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:47 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:30:47.436 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:47 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:30:48.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:48.034+0000 7f3a35b81700 1 -- 172.21.15.29:0/1303430657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a301067d0 msgr2=0x7f3a30106bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:48.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:48.034+0000 7f3a35b81700 1 --2- 172.21.15.29:0/1303430657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a301067d0 0x7f3a30106bb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3a20009910 tx=0x7f3a20009c20 comp rx=0 tx=0).stop 2024-01-24T13:30:48.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:48.034+0000 7f3a35b81700 1 -- 172.21.15.29:0/1303430657 shutdown_connections 2024-01-24T13:30:48.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:48.034+0000 7f3a35b81700 1 --2- 172.21.15.29:0/1303430657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a301067d0 0x7f3a30106bb0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:48.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:48.034+0000 7f3a35b81700 1 -- 172.21.15.29:0/1303430657 >> 172.21.15.29:0/1303430657 conn(0x7f3a30075240 msgr2=0x7f3a30075650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:48.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:48.034+0000 7f3a35b81700 1 -- 172.21.15.29:0/1303430657 shutdown_connections 2024-01-24T13:30:48.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:48.034+0000 7f3a35b81700 1 -- 172.21.15.29:0/1303430657 wait complete. 2024-01-24T13:30:48.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:48.035+0000 7f3a35b81700 1 Processor -- start 2024-01-24T13:30:48.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:48.035+0000 7f3a35b81700 1 -- start start 2024-01-24T13:30:48.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:48.035+0000 7f3a35b81700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a301067d0 0x7f3a3019ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:48.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:48.035+0000 7f3a35b81700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3019bf80 con 0x7f3a301067d0 2024-01-24T13:30:48.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:48.036+0000 7f3a2f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a301067d0 0x7f3a3019ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:48.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:48.036+0000 7f3a2f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a301067d0 0x7f3a3019ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47020/0 (socket says 172.21.15.29:47020) 2024-01-24T13:30:48.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:48.036+0000 7f3a2f7fe700 1 -- 172.21.15.29:0/589597823 learned_addr learned my addr 172.21.15.29:0/589597823 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:48.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:48.036+0000 7f3a2f7fe700 1 -- 172.21.15.29:0/589597823 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a200095f0 con 0x7f3a301067d0 2024-01-24T13:30:48.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:48.036+0000 7f3a2f7fe700 1 --2- 172.21.15.29:0/589597823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a301067d0 0x7f3a3019ba40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3a20004f40 tx=0x7f3a20005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:48.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:48.036+0000 7f3a2cff9700 1 -- 172.21.15.29:0/589597823 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3a2001d070 con 0x7f3a301067d0 2024-01-24T13:30:48.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:48.036+0000 7f3a2cff9700 1 -- 172.21.15.29:0/589597823 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3a20004370 con 0x7f3a301067d0 2024-01-24T13:30:48.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:48.036+0000 7f3a2cff9700 1 -- 172.21.15.29:0/589597823 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3a20003970 con 0x7f3a301067d0 2024-01-24T13:30:48.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:48.036+0000 7f3a35b81700 1 -- 172.21.15.29:0/589597823 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a3019c1e0 con 0x7f3a301067d0 2024-01-24T13:30:48.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:48.036+0000 7f3a35b81700 1 -- 172.21.15.29:0/589597823 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a30195ac0 con 0x7f3a301067d0 2024-01-24T13:30:48.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:48.037+0000 7f3a2cff9700 1 -- 172.21.15.29:0/589597823 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45324+0+0 (secure 0 0 0) 0x7f3a20003be0 con 0x7f3a301067d0 2024-01-24T13:30:48.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:48.037+0000 7f3a2cff9700 1 --2- 172.21.15.29:0/589597823 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f3a180383f0 0x7f3a1803a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:48.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:48.037+0000 7f3a35b81700 1 -- 172.21.15.29:0/589597823 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a30062370 con 0x7f3a301067d0 2024-01-24T13:30:48.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:48.038+0000 7f3a2cff9700 1 -- 172.21.15.29:0/589597823 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3a2004c140 con 0x7f3a301067d0 2024-01-24T13:30:48.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:48.038+0000 7f3a2effd700 1 --2- 172.21.15.29:0/589597823 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f3a180383f0 0x7f3a1803a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:48.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:48.038+0000 7f3a2effd700 1 --2- 172.21.15.29:0/589597823 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f3a180383f0 0x7f3a1803a8b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3a24006fd0 tx=0x7f3a24006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:48.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:48.043+0000 7f3a2cff9700 1 -- 172.21.15.29:0/589597823 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f3a20026070 con 0x7f3a301067d0 2024-01-24T13:30:48.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:48.255+0000 7f3a35b81700 1 -- 172.21.15.29:0/589597823 --> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3a30195cb0 con 0x7f3a180383f0 2024-01-24T13:30:48.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:48.257+0000 7f3a2cff9700 1 -- 172.21.15.29:0/589597823 <== mgr.14152 v2:172.21.15.29:6800/2498991338 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f3a30195cb0 con 0x7f3a180383f0 2024-01-24T13:30:48.258 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:30:48.258 INFO:teuthology.orchestra.run.smithi029.stdout:[{"addr": "172.21.15.29", "hostname": "smithi029", "labels": [], "status": ""}, {"addr": "172.21.15.163", "hostname": "smithi163", "labels": [], "status": ""}] 2024-01-24T13:30:48.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:48.259+0000 7f3a35b81700 1 -- 172.21.15.29:0/589597823 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f3a180383f0 msgr2=0x7f3a1803a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:48.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:48.259+0000 7f3a35b81700 1 --2- 172.21.15.29:0/589597823 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f3a180383f0 0x7f3a1803a8b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3a24006fd0 tx=0x7f3a24006e40 comp rx=0 tx=0).stop 2024-01-24T13:30:48.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:48.259+0000 7f3a35b81700 1 -- 172.21.15.29:0/589597823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a301067d0 msgr2=0x7f3a3019ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:48.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:48.259+0000 7f3a35b81700 1 --2- 172.21.15.29:0/589597823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a301067d0 0x7f3a3019ba40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3a20004f40 tx=0x7f3a20005020 comp rx=0 tx=0).stop 2024-01-24T13:30:48.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:48.259+0000 7f3a35b81700 1 -- 172.21.15.29:0/589597823 shutdown_connections 2024-01-24T13:30:48.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:48.259+0000 7f3a35b81700 1 --2- 172.21.15.29:0/589597823 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f3a180383f0 0x7f3a1803a8b0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:48.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:48.259+0000 7f3a35b81700 1 --2- 172.21.15.29:0/589597823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a301067d0 0x7f3a3019ba40 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:48.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:48.259+0000 7f3a35b81700 1 -- 172.21.15.29:0/589597823 >> 172.21.15.29:0/589597823 conn(0x7f3a30075240 msgr2=0x7f3a300fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:48.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:48.259+0000 7f3a35b81700 1 -- 172.21.15.29:0/589597823 shutdown_connections 2024-01-24T13:30:48.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:48.259+0000 7f3a35b81700 1 -- 172.21.15.29:0/589597823 wait complete. 2024-01-24T13:30:48.632 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi187 2024-01-24T13:30:48.632 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-01-24T13:30:48.632 DEBUG:teuthology.orchestra.run.smithi187:> dd of=/etc/ceph/ceph.conf 2024-01-24T13:30:48.653 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-01-24T13:30:48.653 DEBUG:teuthology.orchestra.run.smithi187:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-24T13:30:48.713 INFO:tasks.cephadm:Adding host smithi187 to orchestrator... 2024-01-24T13:30:48.713 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph orch host add smithi187 2024-01-24T13:30:48.968 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T13:30:49.085 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:48 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:30:49.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:49.918+0000 7ff189fce700 1 -- 172.21.15.29:0/2919354944 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff184105730 msgr2=0x7ff184105b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:49.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:49.918+0000 7ff189fce700 1 --2- 172.21.15.29:0/2919354944 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff184105730 0x7ff184105b10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff174009910 tx=0x7ff174009c20 comp rx=0 tx=0).stop 2024-01-24T13:30:49.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:49.918+0000 7ff189fce700 1 -- 172.21.15.29:0/2919354944 shutdown_connections 2024-01-24T13:30:49.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:49.918+0000 7ff189fce700 1 --2- 172.21.15.29:0/2919354944 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff184105730 0x7ff184105b10 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:49.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:49.918+0000 7ff189fce700 1 -- 172.21.15.29:0/2919354944 >> 172.21.15.29:0/2919354944 conn(0x7ff184075240 msgr2=0x7ff184075650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:49.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:49.918+0000 7ff189fce700 1 -- 172.21.15.29:0/2919354944 shutdown_connections 2024-01-24T13:30:49.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:49.918+0000 7ff189fce700 1 -- 172.21.15.29:0/2919354944 wait complete. 2024-01-24T13:30:49.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:49.919+0000 7ff189fce700 1 Processor -- start 2024-01-24T13:30:49.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:49.919+0000 7ff189fce700 1 -- start start 2024-01-24T13:30:49.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:49.920+0000 7ff189fce700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff184105730 0x7ff184196b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:49.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:49.920+0000 7ff189fce700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff184197090 con 0x7ff184105730 2024-01-24T13:30:49.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:49.920+0000 7ff1837fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff184105730 0x7ff184196b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:49.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:49.920+0000 7ff1837fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff184105730 0x7ff184196b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47028/0 (socket says 172.21.15.29:47028) 2024-01-24T13:30:49.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:49.920+0000 7ff1837fe700 1 -- 172.21.15.29:0/3331976668 learned_addr learned my addr 172.21.15.29:0/3331976668 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:49.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:49.920+0000 7ff1837fe700 1 -- 172.21.15.29:0/3331976668 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1740095f0 con 0x7ff184105730 2024-01-24T13:30:49.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:49.920+0000 7ff1837fe700 1 --2- 172.21.15.29:0/3331976668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff184105730 0x7ff184196b50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff174005850 tx=0x7ff174004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:49.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:49.920+0000 7ff180ff9700 1 -- 172.21.15.29:0/3331976668 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff17401d070 con 0x7ff184105730 2024-01-24T13:30:49.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:49.920+0000 7ff180ff9700 1 -- 172.21.15.29:0/3331976668 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff174004370 con 0x7ff184105730 2024-01-24T13:30:49.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:49.920+0000 7ff189fce700 1 -- 172.21.15.29:0/3331976668 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff184197290 con 0x7ff184105730 2024-01-24T13:30:49.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:49.920+0000 7ff180ff9700 1 -- 172.21.15.29:0/3331976668 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff174003ab0 con 0x7ff184105730 2024-01-24T13:30:49.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:49.920+0000 7ff189fce700 1 -- 172.21.15.29:0/3331976668 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1841976b0 con 0x7ff184105730 2024-01-24T13:30:49.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:49.921+0000 7ff180ff9700 1 -- 172.21.15.29:0/3331976668 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45324+0+0 (secure 0 0 0) 0x7ff17400b6f0 con 0x7ff184105730 2024-01-24T13:30:49.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:49.921+0000 7ff189fce700 1 -- 172.21.15.29:0/3331976668 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff184103970 con 0x7ff184105730 2024-01-24T13:30:49.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:49.922+0000 7ff180ff9700 1 --2- 172.21.15.29:0/3331976668 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7ff1700383a0 0x7ff17003a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:49.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:49.922+0000 7ff180ff9700 1 -- 172.21.15.29:0/3331976668 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff17404c080 con 0x7ff184105730 2024-01-24T13:30:49.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:49.922+0000 7ff182ffd700 1 --2- 172.21.15.29:0/3331976668 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7ff1700383a0 0x7ff17003a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:49.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:49.922+0000 7ff182ffd700 1 --2- 172.21.15.29:0/3331976668 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7ff1700383a0 0x7ff17003a860 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff16c006fd0 tx=0x7ff16c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:49.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:49.927+0000 7ff180ff9700 1 -- 172.21.15.29:0/3331976668 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7ff174005c00 con 0x7ff184105730 2024-01-24T13:30:50.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:49 smithi029 ceph-mon[40651]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-24T13:30:50.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:50.137+0000 7ff189fce700 1 -- 172.21.15.29:0/3331976668 --> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi187", "target": ["mon-mgr", ""]}) v1 -- 0x7ff184068530 con 0x7ff1700383a0 2024-01-24T13:30:51.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:51 smithi029 ceph-mon[40651]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi187", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:30:51.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:51 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:30:51.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:51 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:30:51.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:51 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:30:51.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:51 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:30:51.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:51 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi163", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:30:51.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:51 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:30:51.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:51 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:30:52.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:52.453+0000 7ff180ff9700 1 -- 172.21.15.29:0/3331976668 <== mgr.14152 v2:172.21.15.29:6800/2498991338 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7ff184068530 con 0x7ff1700383a0 2024-01-24T13:30:52.454 INFO:teuthology.orchestra.run.smithi029.stdout:Added host 'smithi187' with addr '172.21.15.187' 2024-01-24T13:30:52.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:52.455+0000 7ff189fce700 1 -- 172.21.15.29:0/3331976668 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7ff1700383a0 msgr2=0x7ff17003a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:52.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:52.455+0000 7ff189fce700 1 --2- 172.21.15.29:0/3331976668 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7ff1700383a0 0x7ff17003a860 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff16c006fd0 tx=0x7ff16c006e40 comp rx=0 tx=0).stop 2024-01-24T13:30:52.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:52.456+0000 7ff189fce700 1 -- 172.21.15.29:0/3331976668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff184105730 msgr2=0x7ff184196b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:52.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:52.456+0000 7ff189fce700 1 --2- 172.21.15.29:0/3331976668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff184105730 0x7ff184196b50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff174005850 tx=0x7ff174004ff0 comp rx=0 tx=0).stop 2024-01-24T13:30:52.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:52.456+0000 7ff189fce700 1 -- 172.21.15.29:0/3331976668 shutdown_connections 2024-01-24T13:30:52.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:52.456+0000 7ff189fce700 1 --2- 172.21.15.29:0/3331976668 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7ff1700383a0 0x7ff17003a860 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:52.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:52.456+0000 7ff189fce700 1 --2- 172.21.15.29:0/3331976668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff184105730 0x7ff184196b50 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:52.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:52.456+0000 7ff189fce700 1 -- 172.21.15.29:0/3331976668 >> 172.21.15.29:0/3331976668 conn(0x7ff184075240 msgr2=0x7ff1840fef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:52.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:52.456+0000 7ff189fce700 1 -- 172.21.15.29:0/3331976668 shutdown_connections 2024-01-24T13:30:52.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:52.456+0000 7ff189fce700 1 -- 172.21.15.29:0/3331976668 wait complete. 2024-01-24T13:30:52.484 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:52 smithi029 ceph-mon[40651]: Updating smithi163:/etc/ceph/ceph.conf 2024-01-24T13:30:52.484 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:52 smithi029 ceph-mon[40651]: Updating smithi163:/var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-24T13:30:52.484 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:52 smithi029 ceph-mon[40651]: Deploying cephadm binary to smithi187 2024-01-24T13:30:52.485 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:52 smithi029 ceph-mon[40651]: Updating smithi163:/etc/ceph/ceph.client.admin.keyring 2024-01-24T13:30:52.485 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:52 smithi029 ceph-mon[40651]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:30:52.485 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:52 smithi029 ceph-mon[40651]: Updating smithi163:/var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/config/ceph.client.admin.keyring 2024-01-24T13:30:52.485 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:52 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:30:52.485 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:52 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:30:52.485 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:52 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:30:52.840 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph orch host ls --format=json 2024-01-24T13:30:53.092 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T13:30:53.453 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:53 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:30:53.453 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:53 smithi029 ceph-mon[40651]: Added host smithi187 2024-01-24T13:30:53.454 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:53 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:30:53.454 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:53 smithi029 ceph-mon[40651]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:30:53.454 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:53 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:30:53.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:53 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:30:54.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:54.058+0000 7f074c52f700 1 -- 172.21.15.29:0/3580843548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07441056b0 msgr2=0x7f0744105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:54.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:54.058+0000 7f074c52f700 1 --2- 172.21.15.29:0/3580843548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07441056b0 0x7f0744105a90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0740009910 tx=0x7f0740009c20 comp rx=0 tx=0).stop 2024-01-24T13:30:54.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:54.059+0000 7f074c52f700 1 -- 172.21.15.29:0/3580843548 shutdown_connections 2024-01-24T13:30:54.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:54.059+0000 7f074c52f700 1 --2- 172.21.15.29:0/3580843548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07441056b0 0x7f0744105a90 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:54.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:54.059+0000 7f074c52f700 1 -- 172.21.15.29:0/3580843548 >> 172.21.15.29:0/3580843548 conn(0x7f07440fb1c0 msgr2=0x7f07440fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:54.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:54.059+0000 7f074c52f700 1 -- 172.21.15.29:0/3580843548 shutdown_connections 2024-01-24T13:30:54.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:54.059+0000 7f074c52f700 1 -- 172.21.15.29:0/3580843548 wait complete. 2024-01-24T13:30:54.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:54.059+0000 7f074c52f700 1 Processor -- start 2024-01-24T13:30:54.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:54.059+0000 7f074c52f700 1 -- start start 2024-01-24T13:30:54.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:54.060+0000 7f074c52f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07441056b0 0x7f0744195cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:54.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:54.060+0000 7f074c52f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0744196200 con 0x7f07441056b0 2024-01-24T13:30:54.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:54.060+0000 7f074a2cb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07441056b0 0x7f0744195cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:54.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:54.060+0000 7f074a2cb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07441056b0 0x7f0744195cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33106/0 (socket says 172.21.15.29:33106) 2024-01-24T13:30:54.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:54.060+0000 7f074a2cb700 1 -- 172.21.15.29:0/4128082155 learned_addr learned my addr 172.21.15.29:0/4128082155 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:54.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:54.060+0000 7f074a2cb700 1 -- 172.21.15.29:0/4128082155 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07400095f0 con 0x7f07441056b0 2024-01-24T13:30:54.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:54.060+0000 7f074a2cb700 1 --2- 172.21.15.29:0/4128082155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07441056b0 0x7f0744195cc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0740004f40 tx=0x7f0740005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:54.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:54.060+0000 7f073b7fe700 1 -- 172.21.15.29:0/4128082155 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f074001d070 con 0x7f07441056b0 2024-01-24T13:30:54.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:54.060+0000 7f074c52f700 1 -- 172.21.15.29:0/4128082155 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0744196460 con 0x7f07441056b0 2024-01-24T13:30:54.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:54.060+0000 7f073b7fe700 1 -- 172.21.15.29:0/4128082155 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0740004370 con 0x7f07441056b0 2024-01-24T13:30:54.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:54.060+0000 7f073b7fe700 1 -- 172.21.15.29:0/4128082155 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0740003970 con 0x7f07441056b0 2024-01-24T13:30:54.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:54.061+0000 7f074c52f700 1 -- 172.21.15.29:0/4128082155 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07441968e0 con 0x7f07441056b0 2024-01-24T13:30:54.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:54.061+0000 7f073b7fe700 1 -- 172.21.15.29:0/4128082155 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45324+0+0 (secure 0 0 0) 0x7f0740003c30 con 0x7f07441056b0 2024-01-24T13:30:54.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:54.061+0000 7f074c52f700 1 -- 172.21.15.29:0/4128082155 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0744062370 con 0x7f07441056b0 2024-01-24T13:30:54.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:54.062+0000 7f073b7fe700 1 --2- 172.21.15.29:0/4128082155 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f07300383a0 0x7f073003a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:54.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:54.062+0000 7f073b7fe700 1 -- 172.21.15.29:0/4128082155 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f074004c070 con 0x7f07441056b0 2024-01-24T13:30:54.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:54.062+0000 7f0749aca700 1 --2- 172.21.15.29:0/4128082155 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f07300383a0 0x7f073003a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:54.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:54.062+0000 7f0749aca700 1 --2- 172.21.15.29:0/4128082155 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f07300383a0 0x7f073003a860 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0734006fd0 tx=0x7f0734006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:54.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:54.067+0000 7f073b7fe700 1 -- 172.21.15.29:0/4128082155 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f074007f120 con 0x7f07441056b0 2024-01-24T13:30:54.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:54.280+0000 7f074c52f700 1 -- 172.21.15.29:0/4128082155 --> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0744068ee0 con 0x7f07300383a0 2024-01-24T13:30:54.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:54.282+0000 7f073b7fe700 1 -- 172.21.15.29:0/4128082155 <== mgr.14152 v2:172.21.15.29:6800/2498991338 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+240 (secure 0 0 0) 0x7f0744068ee0 con 0x7f07300383a0 2024-01-24T13:30:54.283 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:30:54.283 INFO:teuthology.orchestra.run.smithi029.stdout:[{"addr": "172.21.15.29", "hostname": "smithi029", "labels": [], "status": ""}, {"addr": "172.21.15.163", "hostname": "smithi163", "labels": [], "status": ""}, {"addr": "172.21.15.187", "hostname": "smithi187", "labels": [], "status": ""}] 2024-01-24T13:30:54.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:54.284+0000 7f074c52f700 1 -- 172.21.15.29:0/4128082155 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f07300383a0 msgr2=0x7f073003a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:54.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:54.284+0000 7f074c52f700 1 --2- 172.21.15.29:0/4128082155 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f07300383a0 0x7f073003a860 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0734006fd0 tx=0x7f0734006e40 comp rx=0 tx=0).stop 2024-01-24T13:30:54.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:54.284+0000 7f074c52f700 1 -- 172.21.15.29:0/4128082155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07441056b0 msgr2=0x7f0744195cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:54.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:54.284+0000 7f074c52f700 1 --2- 172.21.15.29:0/4128082155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07441056b0 0x7f0744195cc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0740004f40 tx=0x7f0740005020 comp rx=0 tx=0).stop 2024-01-24T13:30:54.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:54.284+0000 7f074c52f700 1 -- 172.21.15.29:0/4128082155 shutdown_connections 2024-01-24T13:30:54.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:54.284+0000 7f074c52f700 1 --2- 172.21.15.29:0/4128082155 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f07300383a0 0x7f073003a860 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:54.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:54.284+0000 7f074c52f700 1 --2- 172.21.15.29:0/4128082155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07441056b0 0x7f0744195cc0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:54.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:54.284+0000 7f074c52f700 1 -- 172.21.15.29:0/4128082155 >> 172.21.15.29:0/4128082155 conn(0x7f07440fb1c0 msgr2=0x7f07440687d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:54.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:54.284+0000 7f074c52f700 1 -- 172.21.15.29:0/4128082155 shutdown_connections 2024-01-24T13:30:54.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:54.284+0000 7f074c52f700 1 -- 172.21.15.29:0/4128082155 wait complete. 2024-01-24T13:30:54.664 INFO:tasks.cephadm:Setting crush tunables to default 2024-01-24T13:30:54.665 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph osd crush tunables default 2024-01-24T13:30:54.913 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T13:30:55.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:54 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:30:55.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:55.893+0000 7feac318a700 1 -- 172.21.15.29:0/3655457334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feabc0ff0a0 msgr2=0x7feabc0ff480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:55.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:55.893+0000 7feac318a700 1 --2- 172.21.15.29:0/3655457334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feabc0ff0a0 0x7feabc0ff480 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7feaac009910 tx=0x7feaac009c20 comp rx=0 tx=0).stop 2024-01-24T13:30:55.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:55.893+0000 7feac318a700 1 -- 172.21.15.29:0/3655457334 shutdown_connections 2024-01-24T13:30:55.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:55.893+0000 7feac318a700 1 --2- 172.21.15.29:0/3655457334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feabc0ff0a0 0x7feabc0ff480 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:55.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:55.893+0000 7feac318a700 1 -- 172.21.15.29:0/3655457334 >> 172.21.15.29:0/3655457334 conn(0x7feabc075da0 msgr2=0x7feabc0761b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:55.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:55.893+0000 7feac318a700 1 -- 172.21.15.29:0/3655457334 shutdown_connections 2024-01-24T13:30:55.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:55.893+0000 7feac318a700 1 -- 172.21.15.29:0/3655457334 wait complete. 2024-01-24T13:30:55.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:55.894+0000 7feac318a700 1 Processor -- start 2024-01-24T13:30:55.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:55.894+0000 7feac318a700 1 -- start start 2024-01-24T13:30:55.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:55.894+0000 7feac318a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feabc0ff0a0 0x7feabc195d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:55.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:55.894+0000 7feac318a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feabc196250 con 0x7feabc0ff0a0 2024-01-24T13:30:55.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:55.894+0000 7feac0f26700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feabc0ff0a0 0x7feabc195d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:55.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:55.894+0000 7feac0f26700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feabc0ff0a0 0x7feabc195d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33126/0 (socket says 172.21.15.29:33126) 2024-01-24T13:30:55.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:55.894+0000 7feac0f26700 1 -- 172.21.15.29:0/3350081638 learned_addr learned my addr 172.21.15.29:0/3350081638 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:30:55.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:55.895+0000 7feac0f26700 1 -- 172.21.15.29:0/3350081638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaac0095f0 con 0x7feabc0ff0a0 2024-01-24T13:30:55.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:55.895+0000 7feac0f26700 1 --2- 172.21.15.29:0/3350081638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feabc0ff0a0 0x7feabc195d10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7feaac004f40 tx=0x7feaac005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:55.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:55.895+0000 7feab9ffb700 1 -- 172.21.15.29:0/3350081638 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feaac01d070 con 0x7feabc0ff0a0 2024-01-24T13:30:55.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:55.895+0000 7feab9ffb700 1 -- 172.21.15.29:0/3350081638 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feaac004370 con 0x7feabc0ff0a0 2024-01-24T13:30:55.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:55.895+0000 7feac318a700 1 -- 172.21.15.29:0/3350081638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feabc196450 con 0x7feabc0ff0a0 2024-01-24T13:30:55.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:55.895+0000 7feab9ffb700 1 -- 172.21.15.29:0/3350081638 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feaac0039b0 con 0x7feabc0ff0a0 2024-01-24T13:30:55.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:55.895+0000 7feac318a700 1 -- 172.21.15.29:0/3350081638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feabc196870 con 0x7feabc0ff0a0 2024-01-24T13:30:55.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:55.896+0000 7feab9ffb700 1 -- 172.21.15.29:0/3350081638 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45324+0+0 (secure 0 0 0) 0x7feaac003c20 con 0x7feabc0ff0a0 2024-01-24T13:30:55.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:55.896+0000 7feac318a700 1 -- 172.21.15.29:0/3350081638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feabc062370 con 0x7feabc0ff0a0 2024-01-24T13:30:55.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:55.896+0000 7feab9ffb700 1 --2- 172.21.15.29:0/3350081638 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7feaa40383f0 0x7feaa403a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:55.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:55.896+0000 7feab9ffb700 1 -- 172.21.15.29:0/3350081638 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7feaac04c300 con 0x7feabc0ff0a0 2024-01-24T13:30:55.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:55.896+0000 7feabbfff700 1 --2- 172.21.15.29:0/3350081638 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7feaa40383f0 0x7feaa403a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:55.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:55.897+0000 7feabbfff700 1 --2- 172.21.15.29:0/3350081638 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7feaa40383f0 0x7feaa403a8b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7feab0006fd0 tx=0x7feab0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:55.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:55.902+0000 7feab9ffb700 1 -- 172.21.15.29:0/3350081638 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7feaac029950 con 0x7feabc0ff0a0 2024-01-24T13:30:56.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:56.120+0000 7feac318a700 1 -- 172.21.15.29:0/3350081638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7feabc18f950 con 0x7feabc0ff0a0 2024-01-24T13:30:56.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:56 smithi029 ceph-mon[40651]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-24T13:30:56.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:56 smithi029 ceph-mon[40651]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:30:57.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:57.022+0000 7feab9ffb700 1 -- 172.21.15.29:0/3350081638 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7feaac026070 con 0x7feabc0ff0a0 2024-01-24T13:30:57.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:57.025+0000 7feac318a700 1 -- 172.21.15.29:0/3350081638 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7feaa40383f0 msgr2=0x7feaa403a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:57.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:57.025+0000 7feac318a700 1 --2- 172.21.15.29:0/3350081638 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7feaa40383f0 0x7feaa403a8b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7feab0006fd0 tx=0x7feab0006e40 comp rx=0 tx=0).stop 2024-01-24T13:30:57.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:57.025+0000 7feac318a700 1 -- 172.21.15.29:0/3350081638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feabc0ff0a0 msgr2=0x7feabc195d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:57.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:57.025+0000 7feac318a700 1 --2- 172.21.15.29:0/3350081638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feabc0ff0a0 0x7feabc195d10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7feaac004f40 tx=0x7feaac005020 comp rx=0 tx=0).stop 2024-01-24T13:30:57.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:57.025+0000 7feac318a700 1 -- 172.21.15.29:0/3350081638 shutdown_connections 2024-01-24T13:30:57.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:57.025+0000 7feac318a700 1 --2- 172.21.15.29:0/3350081638 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7feaa40383f0 0x7feaa403a8b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:57.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:57.025+0000 7feac318a700 1 --2- 172.21.15.29:0/3350081638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feabc0ff0a0 0x7feabc195d10 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:57.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:57.025+0000 7feac318a700 1 -- 172.21.15.29:0/3350081638 >> 172.21.15.29:0/3350081638 conn(0x7feabc075da0 msgr2=0x7feabc106f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:57.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:57.026+0000 7feac318a700 1 -- 172.21.15.29:0/3350081638 shutdown_connections 2024-01-24T13:30:57.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:30:57.026+0000 7feac318a700 1 -- 172.21.15.29:0/3350081638 wait complete. 2024-01-24T13:30:57.027 INFO:teuthology.orchestra.run.smithi029.stderr:adjusted tunables profile to default 2024-01-24T13:30:57.330 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:57 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:30:57.331 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:57 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:30:57.331 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:57 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:30:57.331 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:57 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:30:57.331 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:57 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi187", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:30:57.331 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:57 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:30:57.331 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:57 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:30:57.331 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:57 smithi029 ceph-mon[40651]: Updating smithi187:/etc/ceph/ceph.conf 2024-01-24T13:30:57.331 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:57 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/3350081638' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-01-24T13:30:57.331 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:57 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:30:57.331 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:57 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:30:57.331 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:57 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:30:57.412 INFO:tasks.cephadm:Adding mon.a on smithi029 2024-01-24T13:30:57.412 INFO:tasks.cephadm:Adding mon.b on smithi163 2024-01-24T13:30:57.413 INFO:tasks.cephadm:Adding mon.c on smithi187 2024-01-24T13:30:57.413 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph orch apply mon '3;smithi029:172.21.15.29=a;smithi163:172.21.15.163=b;smithi187:172.21.15.187=c' 2024-01-24T13:30:57.665 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-24T13:30:58.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:58 smithi029 ceph-mon[40651]: Updating smithi187:/var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-24T13:30:58.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:58 smithi029 ceph-mon[40651]: Updating smithi187:/etc/ceph/ceph.client.admin.keyring 2024-01-24T13:30:58.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:58 smithi029 ceph-mon[40651]: Updating smithi187:/var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/config/ceph.client.admin.keyring 2024-01-24T13:30:58.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:58 smithi029 ceph-mon[40651]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:30:58.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:58 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/3350081638' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-01-24T13:30:58.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:58 smithi029 ceph-mon[40651]: osdmap e4: 0 total, 0 up, 0 in 2024-01-24T13:30:58.600 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:30:58.599+0000 7f0bb95b7700 1 -- 172.21.15.187:0/3381013971 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bb4073060 msgr2=0x7f0bb4073440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:58.600 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:30:58.599+0000 7f0bb95b7700 1 --2- 172.21.15.187:0/3381013971 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bb4073060 0x7f0bb4073440 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8009910 tx=0x7f0ba8009c20 comp rx=0 tx=0).stop 2024-01-24T13:30:58.600 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:30:58.600+0000 7f0bb95b7700 1 -- 172.21.15.187:0/3381013971 shutdown_connections 2024-01-24T13:30:58.600 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:30:58.600+0000 7f0bb95b7700 1 --2- 172.21.15.187:0/3381013971 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bb4073060 0x7f0bb4073440 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:58.600 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:30:58.600+0000 7f0bb95b7700 1 -- 172.21.15.187:0/3381013971 >> 172.21.15.187:0/3381013971 conn(0x7f0bb40fc2d0 msgr2=0x7f0bb40fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:58.601 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:30:58.600+0000 7f0bb95b7700 1 -- 172.21.15.187:0/3381013971 shutdown_connections 2024-01-24T13:30:58.601 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:30:58.600+0000 7f0bb95b7700 1 -- 172.21.15.187:0/3381013971 wait complete. 2024-01-24T13:30:58.601 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:30:58.600+0000 7f0bb95b7700 1 Processor -- start 2024-01-24T13:30:58.601 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:30:58.600+0000 7f0bb95b7700 1 -- start start 2024-01-24T13:30:58.601 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:30:58.601+0000 7f0bb95b7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bb4073060 0x7f0bb4197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:58.601 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:30:58.601+0000 7f0bb95b7700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb4198480 con 0x7f0bb4073060 2024-01-24T13:30:58.603 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:30:58.601+0000 7f0bb2ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bb4073060 0x7f0bb4197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:58.603 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:30:58.601+0000 7f0bb2ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bb4073060 0x7f0bb4197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.187:58108/0 (socket says 172.21.15.187:58108) 2024-01-24T13:30:58.603 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:30:58.601+0000 7f0bb2ffd700 1 -- 172.21.15.187:0/1318826562 learned_addr learned my addr 172.21.15.187:0/1318826562 (peer_addr_for_me v2:172.21.15.187:0/0) 2024-01-24T13:30:58.603 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:30:58.601+0000 7f0bb2ffd700 1 -- 172.21.15.187:0/1318826562 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba80095f0 con 0x7f0bb4073060 2024-01-24T13:30:58.604 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:30:58.601+0000 7f0bb2ffd700 1 --2- 172.21.15.187:0/1318826562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bb4073060 0x7f0bb4197f40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8006010 tx=0x7f0ba8004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:58.604 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:30:58.602+0000 7f0b9bfff700 1 -- 172.21.15.187:0/1318826562 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ba801d070 con 0x7f0bb4073060 2024-01-24T13:30:58.604 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:30:58.602+0000 7f0bb95b7700 1 -- 172.21.15.187:0/1318826562 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bb4198680 con 0x7f0bb4073060 2024-01-24T13:30:58.604 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:30:58.602+0000 7f0b9bfff700 1 -- 172.21.15.187:0/1318826562 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0ba80044a0 con 0x7f0bb4073060 2024-01-24T13:30:58.604 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:30:58.602+0000 7f0b9bfff700 1 -- 172.21.15.187:0/1318826562 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ba8003a70 con 0x7f0bb4073060 2024-01-24T13:30:58.604 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:30:58.602+0000 7f0bb95b7700 1 -- 172.21.15.187:0/1318826562 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bb4198aa0 con 0x7f0bb4073060 2024-01-24T13:30:58.604 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:30:58.603+0000 7f0b9bfff700 1 -- 172.21.15.187:0/1318826562 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45324+0+0 (secure 0 0 0) 0x7f0ba8003c60 con 0x7f0bb4073060 2024-01-24T13:30:58.604 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:30:58.603+0000 7f0bb95b7700 1 -- 172.21.15.187:0/1318826562 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bb4062370 con 0x7f0bb4073060 2024-01-24T13:30:58.604 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:30:58.603+0000 7f0b9bfff700 1 --2- 172.21.15.187:0/1318826562 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f0b9c0383f0 0x7f0b9c03a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:30:58.604 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:30:58.603+0000 7f0b9bfff700 1 -- 172.21.15.187:0/1318826562 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0ba804c0f0 con 0x7f0bb4073060 2024-01-24T13:30:58.604 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:30:58.603+0000 7f0bb27fc700 1 --2- 172.21.15.187:0/1318826562 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f0b9c0383f0 0x7f0b9c03a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:30:58.604 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:30:58.604+0000 7f0bb27fc700 1 --2- 172.21.15.187:0/1318826562 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f0b9c0383f0 0x7f0b9c03a8b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0ba4006fd0 tx=0x7f0ba4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:30:58.607 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:30:58.608+0000 7f0b9bfff700 1 -- 172.21.15.187:0/1318826562 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f0ba8029360 con 0x7f0bb4073060 2024-01-24T13:30:58.804 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:30:58.804+0000 7f0bb95b7700 1 -- 172.21.15.187:0/1318826562 --> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi029:172.21.15.29=a;smithi163:172.21.15.163=b;smithi187:172.21.15.187=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f0bb4109d80 con 0x7f0b9c0383f0 2024-01-24T13:30:58.810 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:30:58.811+0000 7f0b9bfff700 1 -- 172.21.15.187:0/1318826562 <== mgr.14152 v2:172.21.15.29:6800/2498991338 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f0bb4109d80 con 0x7f0b9c0383f0 2024-01-24T13:30:58.810 INFO:teuthology.orchestra.run.smithi187.stdout:Scheduled mon update... 2024-01-24T13:30:58.812 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:30:58.812+0000 7f0bb95b7700 1 -- 172.21.15.187:0/1318826562 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f0b9c0383f0 msgr2=0x7f0b9c03a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:58.812 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:30:58.813+0000 7f0bb95b7700 1 --2- 172.21.15.187:0/1318826562 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f0b9c0383f0 0x7f0b9c03a8b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0ba4006fd0 tx=0x7f0ba4006e40 comp rx=0 tx=0).stop 2024-01-24T13:30:58.812 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:30:58.813+0000 7f0bb95b7700 1 -- 172.21.15.187:0/1318826562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bb4073060 msgr2=0x7f0bb4197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:30:58.812 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:30:58.813+0000 7f0bb95b7700 1 --2- 172.21.15.187:0/1318826562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bb4073060 0x7f0bb4197f40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8006010 tx=0x7f0ba8004ff0 comp rx=0 tx=0).stop 2024-01-24T13:30:58.812 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:30:58.813+0000 7f0bb95b7700 1 -- 172.21.15.187:0/1318826562 shutdown_connections 2024-01-24T13:30:58.813 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:30:58.813+0000 7f0bb95b7700 1 --2- 172.21.15.187:0/1318826562 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f0b9c0383f0 0x7f0b9c03a8b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:58.813 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:30:58.813+0000 7f0bb95b7700 1 --2- 172.21.15.187:0/1318826562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bb4073060 0x7f0bb4197f40 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:30:58.813 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:30:58.813+0000 7f0bb95b7700 1 -- 172.21.15.187:0/1318826562 >> 172.21.15.187:0/1318826562 conn(0x7f0bb40fc2d0 msgr2=0x7f0bb4107380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:30:58.813 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:30:58.813+0000 7f0bb95b7700 1 -- 172.21.15.187:0/1318826562 shutdown_connections 2024-01-24T13:30:58.813 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:30:58.813+0000 7f0bb95b7700 1 -- 172.21.15.187:0/1318826562 wait complete. 2024-01-24T13:30:59.190 DEBUG:teuthology.orchestra.run.smithi163:mon.b> sudo journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mon.b.service 2024-01-24T13:30:59.193 DEBUG:teuthology.orchestra.run.smithi187:mon.c> sudo journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mon.c.service 2024-01-24T13:30:59.195 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-01-24T13:30:59.196 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph mon dump -f json 2024-01-24T13:30:59.227 INFO:journalctl@ceph.mon.c.smithi187.stdout:-- Logs begin at Wed 2024-01-24 13:13:23 UTC. -- 2024-01-24T13:30:59.230 INFO:journalctl@ceph.mon.b.smithi163.stdout:-- Logs begin at Wed 2024-01-24 13:14:09 UTC. -- 2024-01-24T13:30:59.544 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-24T13:31:00.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:59 smithi029 ceph-mon[40651]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:00.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:59 smithi029 ceph-mon[40651]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi029:172.21.15.29=a;smithi163:172.21.15.163=b;smithi187:172.21.15.187=c", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:31:00.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:59 smithi029 ceph-mon[40651]: Saving service mon spec with placement smithi029:172.21.15.29=a;smithi163:172.21.15.163=b;smithi187:172.21.15.187=c;count:3 2024-01-24T13:31:00.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:59 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:00.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:59 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:31:00.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:59 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:00.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:59 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:31:00.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:59 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:00.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:59 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-24T13:31:00.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:59 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:00.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:30:59 smithi029 ceph-mon[40651]: Deploying daemon mon.c on smithi187 2024-01-24T13:31:01.314 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:01.313+0000 7fd07da6e700 1 -- 172.21.15.187:0/719340580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0781067d0 msgr2=0x7fd078106bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:31:01.314 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:01.313+0000 7fd07da6e700 1 --2- 172.21.15.187:0/719340580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0781067d0 0x7fd078106bb0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd068009910 tx=0x7fd068009c20 comp rx=0 tx=0).stop 2024-01-24T13:31:01.314 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:01.313+0000 7fd07da6e700 1 -- 172.21.15.187:0/719340580 shutdown_connections 2024-01-24T13:31:01.314 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:01.313+0000 7fd07da6e700 1 --2- 172.21.15.187:0/719340580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0781067d0 0x7fd078106bb0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:01.314 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:01.313+0000 7fd07da6e700 1 -- 172.21.15.187:0/719340580 >> 172.21.15.187:0/719340580 conn(0x7fd078075240 msgr2=0x7fd078075650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:31:01.314 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:01.313+0000 7fd07da6e700 1 -- 172.21.15.187:0/719340580 shutdown_connections 2024-01-24T13:31:01.315 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:01.313+0000 7fd07da6e700 1 -- 172.21.15.187:0/719340580 wait complete. 2024-01-24T13:31:01.315 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:01.314+0000 7fd07da6e700 1 Processor -- start 2024-01-24T13:31:01.315 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:01.314+0000 7fd07da6e700 1 -- start start 2024-01-24T13:31:01.315 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:01.314+0000 7fd07da6e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0781067d0 0x7fd07819ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:31:01.315 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:01.314+0000 7fd07da6e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07819bf80 con 0x7fd0781067d0 2024-01-24T13:31:01.315 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:01.314+0000 7fd0777fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0781067d0 0x7fd07819ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:31:01.315 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:01.314+0000 7fd0777fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0781067d0 0x7fd07819ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.187:58114/0 (socket says 172.21.15.187:58114) 2024-01-24T13:31:01.315 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:01.314+0000 7fd0777fe700 1 -- 172.21.15.187:0/2497112999 learned_addr learned my addr 172.21.15.187:0/2497112999 (peer_addr_for_me v2:172.21.15.187:0/0) 2024-01-24T13:31:01.315 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:01.315+0000 7fd0777fe700 1 -- 172.21.15.187:0/2497112999 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0680095f0 con 0x7fd0781067d0 2024-01-24T13:31:01.315 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:01.315+0000 7fd0777fe700 1 --2- 172.21.15.187:0/2497112999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0781067d0 0x7fd07819ba40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd068004f40 tx=0x7fd068005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:31:01.316 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:01.315+0000 7fd074ff9700 1 -- 172.21.15.187:0/2497112999 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd06801d070 con 0x7fd0781067d0 2024-01-24T13:31:01.316 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:01.315+0000 7fd07da6e700 1 -- 172.21.15.187:0/2497112999 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd07819c1e0 con 0x7fd0781067d0 2024-01-24T13:31:01.316 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:01.315+0000 7fd074ff9700 1 -- 172.21.15.187:0/2497112999 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd068004370 con 0x7fd0781067d0 2024-01-24T13:31:01.316 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:01.315+0000 7fd07da6e700 1 -- 172.21.15.187:0/2497112999 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd078195ac0 con 0x7fd0781067d0 2024-01-24T13:31:01.317 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:01.315+0000 7fd074ff9700 1 -- 172.21.15.187:0/2497112999 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd068003970 con 0x7fd0781067d0 2024-01-24T13:31:01.317 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:01.316+0000 7fd074ff9700 1 -- 172.21.15.187:0/2497112999 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45324+0+0 (secure 0 0 0) 0x7fd068003c30 con 0x7fd0781067d0 2024-01-24T13:31:01.317 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:01.316+0000 7fd07da6e700 1 -- 172.21.15.187:0/2497112999 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd064005320 con 0x7fd0781067d0 2024-01-24T13:31:01.317 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:01.316+0000 7fd074ff9700 1 --2- 172.21.15.187:0/2497112999 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fd0600383a0 0x7fd06003a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:31:01.317 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:01.316+0000 7fd074ff9700 1 -- 172.21.15.187:0/2497112999 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd06804c0f0 con 0x7fd0781067d0 2024-01-24T13:31:01.317 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:01.316+0000 7fd076ffd700 1 --2- 172.21.15.187:0/2497112999 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fd0600383a0 0x7fd06003a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:31:01.317 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:01.317+0000 7fd076ffd700 1 --2- 172.21.15.187:0/2497112999 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fd0600383a0 0x7fd06003a860 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd06c006fd0 tx=0x7fd06c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:31:01.321 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:01.321+0000 7fd074ff9700 1 -- 172.21.15.187:0/2497112999 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fd06802a590 con 0x7fd0781067d0 2024-01-24T13:31:01.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 systemd[1]: Starting Ceph mon.c for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:31:01.617 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:01.616+0000 7fd07da6e700 1 -- 172.21.15.187:0/2497112999 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd0640059f0 con 0x7fd0781067d0 2024-01-24T13:31:01.617 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:01.617+0000 7fd074ff9700 1 -- 172.21.15.187:0/2497112999 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+736 (secure 0 0 0) 0x7fd068026070 con 0x7fd0781067d0 2024-01-24T13:31:01.617 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-01-24T13:31:01.617 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":1,"fsid":"83afcf5e-babc-11ee-95b1-87774f69a715","modified":"2024-01-24T13:29:31.023487Z","created":"2024-01-24T13:29:31.023487Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-24T13:31:01.619 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:01.620+0000 7fd07da6e700 1 -- 172.21.15.187:0/2497112999 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fd0600383a0 msgr2=0x7fd06003a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:31:01.619 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:01.620+0000 7fd07da6e700 1 --2- 172.21.15.187:0/2497112999 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fd0600383a0 0x7fd06003a860 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd06c006fd0 tx=0x7fd06c006e40 comp rx=0 tx=0).stop 2024-01-24T13:31:01.620 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:01.620+0000 7fd07da6e700 1 -- 172.21.15.187:0/2497112999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0781067d0 msgr2=0x7fd07819ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:31:01.620 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:01.620+0000 7fd07da6e700 1 --2- 172.21.15.187:0/2497112999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0781067d0 0x7fd07819ba40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd068004f40 tx=0x7fd068005020 comp rx=0 tx=0).stop 2024-01-24T13:31:01.620 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:01.620+0000 7fd07da6e700 1 -- 172.21.15.187:0/2497112999 shutdown_connections 2024-01-24T13:31:01.620 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:01.620+0000 7fd07da6e700 1 --2- 172.21.15.187:0/2497112999 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fd0600383a0 0x7fd06003a860 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:01.620 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:01.620+0000 7fd07da6e700 1 --2- 172.21.15.187:0/2497112999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0781067d0 0x7fd07819ba40 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:01.620 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:01.620+0000 7fd07da6e700 1 -- 172.21.15.187:0/2497112999 >> 172.21.15.187:0/2497112999 conn(0x7fd078075240 msgr2=0x7fd0780fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:31:01.620 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:01.620+0000 7fd07da6e700 1 -- 172.21.15.187:0/2497112999 shutdown_connections 2024-01-24T13:31:01.620 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:01.620+0000 7fd07da6e700 1 -- 172.21.15.187:0/2497112999 wait complete. 2024-01-24T13:31:01.620 INFO:teuthology.orchestra.run.smithi187.stderr:dumped monmap epoch 1 2024-01-24T13:31:01.909 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: set uid:gid to 167:167 (ceph:ceph) 2024-01-24T13:31:01.910 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable), process ceph-mon, pid 7 2024-01-24T13:31:01.910 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: pidfile_write: ignore empty --pid-file 2024-01-24T13:31:01.910 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: load: jerasure load: lrc 2024-01-24T13:31:01.910 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: RocksDB version: 7.9.2 2024-01-24T13:31:01.910 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Git sha 0 2024-01-24T13:31:01.910 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Compile date 2024-01-24 06:39:28 2024-01-24T13:31:01.910 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: DB SUMMARY 2024-01-24T13:31:01.910 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: DB Session ID: LXA7L5U3VOR2WJO3VGUA 2024-01-24T13:31:01.910 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: CURRENT file: CURRENT 2024-01-24T13:31:01.910 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: IDENTITY file: IDENTITY 2024-01-24T13:31:01.910 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-01-24T13:31:01.910 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-01-24T13:31:01.910 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-01-24T13:31:01.910 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.error_if_exists: 0 2024-01-24T13:31:01.911 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.create_if_missing: 0 2024-01-24T13:31:01.911 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.paranoid_checks: 1 2024-01-24T13:31:01.911 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.flush_verify_memtable_count: 1 2024-01-24T13:31:01.911 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-01-24T13:31:01.911 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-01-24T13:31:01.911 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.env: 0x5610f4c5f720 2024-01-24T13:31:01.911 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.fs: PosixFileSystem 2024-01-24T13:31:01.911 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.info_log: 0x5610f733b460 2024-01-24T13:31:01.911 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.max_file_opening_threads: 16 2024-01-24T13:31:01.911 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.statistics: (nil) 2024-01-24T13:31:01.911 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.use_fsync: 0 2024-01-24T13:31:01.911 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.max_log_file_size: 0 2024-01-24T13:31:01.911 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-24T13:31:01.911 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.log_file_time_to_roll: 0 2024-01-24T13:31:01.912 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.keep_log_file_num: 1000 2024-01-24T13:31:01.912 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.recycle_log_file_num: 0 2024-01-24T13:31:01.912 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.allow_fallocate: 1 2024-01-24T13:31:01.912 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.allow_mmap_reads: 0 2024-01-24T13:31:01.912 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.allow_mmap_writes: 0 2024-01-24T13:31:01.912 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.use_direct_reads: 0 2024-01-24T13:31:01.912 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-24T13:31:01.912 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.create_missing_column_families: 0 2024-01-24T13:31:01.912 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.db_log_dir: 2024-01-24T13:31:01.912 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.wal_dir: 2024-01-24T13:31:01.912 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.table_cache_numshardbits: 6 2024-01-24T13:31:01.912 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.WAL_ttl_seconds: 0 2024-01-24T13:31:01.912 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.WAL_size_limit_MB: 0 2024-01-24T13:31:01.912 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-24T13:31:01.913 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-24T13:31:01.913 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.is_fd_close_on_exec: 1 2024-01-24T13:31:01.913 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.advise_random_on_open: 1 2024-01-24T13:31:01.913 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.db_write_buffer_size: 0 2024-01-24T13:31:01.913 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.write_buffer_manager: 0x5610f65cc320 2024-01-24T13:31:01.913 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-24T13:31:01.913 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-24T13:31:01.913 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.use_adaptive_mutex: 0 2024-01-24T13:31:01.913 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.rate_limiter: (nil) 2024-01-24T13:31:01.913 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-24T13:31:01.913 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.wal_recovery_mode: 2 2024-01-24T13:31:01.913 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.enable_thread_tracking: 0 2024-01-24T13:31:01.913 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.enable_pipelined_write: 0 2024-01-24T13:31:01.913 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.unordered_write: 0 2024-01-24T13:31:01.913 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-24T13:31:01.914 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-24T13:31:01.914 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-24T13:31:01.914 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-24T13:31:01.914 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.row_cache: None 2024-01-24T13:31:01.914 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.wal_filter: None 2024-01-24T13:31:01.914 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-24T13:31:01.914 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.allow_ingest_behind: 0 2024-01-24T13:31:01.914 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.two_write_queues: 0 2024-01-24T13:31:01.914 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.manual_wal_flush: 0 2024-01-24T13:31:01.914 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.wal_compression: 0 2024-01-24T13:31:01.914 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.atomic_flush: 0 2024-01-24T13:31:01.914 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-24T13:31:01.914 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.persist_stats_to_disk: 0 2024-01-24T13:31:01.914 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.write_dbid_to_manifest: 0 2024-01-24T13:31:01.914 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.log_readahead_size: 0 2024-01-24T13:31:01.914 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-01-24T13:31:01.915 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.best_efforts_recovery: 0 2024-01-24T13:31:01.915 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-01-24T13:31:01.915 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-01-24T13:31:01.915 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.allow_data_in_errors: 0 2024-01-24T13:31:01.915 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.db_host_id: __hostname__ 2024-01-24T13:31:01.915 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.enforce_single_del_contracts: true 2024-01-24T13:31:01.915 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.max_background_jobs: 2 2024-01-24T13:31:01.915 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.max_background_compactions: -1 2024-01-24T13:31:01.915 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.max_subcompactions: 1 2024-01-24T13:31:01.915 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-24T13:31:01.915 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-24T13:31:01.915 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.delayed_write_rate : 16777216 2024-01-24T13:31:01.915 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.max_total_wal_size: 0 2024-01-24T13:31:01.915 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-24T13:31:01.916 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.stats_dump_period_sec: 600 2024-01-24T13:31:01.916 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.stats_persist_period_sec: 600 2024-01-24T13:31:01.916 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-24T13:31:01.916 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.max_open_files: -1 2024-01-24T13:31:01.916 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.bytes_per_sync: 0 2024-01-24T13:31:01.916 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.wal_bytes_per_sync: 0 2024-01-24T13:31:01.916 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.strict_bytes_per_sync: 0 2024-01-24T13:31:01.916 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.compaction_readahead_size: 0 2024-01-24T13:31:01.916 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.max_background_flushes: -1 2024-01-24T13:31:01.916 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Compression algorithms supported: 2024-01-24T13:31:01.916 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-24T13:31:01.916 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: kZSTD supported: 0 2024-01-24T13:31:01.916 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: kXpressCompression supported: 0 2024-01-24T13:31:01.916 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: kLZ4HCCompression supported: 1 2024-01-24T13:31:01.916 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: kZlibCompression supported: 1 2024-01-24T13:31:01.917 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: kSnappyCompression supported: 1 2024-01-24T13:31:01.917 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: kLZ4Compression supported: 1 2024-01-24T13:31:01.917 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: kBZip2Compression supported: 0 2024-01-24T13:31:01.917 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Fast CRC32 supported: Supported on x86 2024-01-24T13:31:01.917 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: DMutex implementation: pthread_mutex_t 2024-01-24T13:31:01.917 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-01-24T13:31:01.917 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-01-24T13:31:01.917 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-24T13:31:01.917 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.merge_operator: 2024-01-24T13:31:01.917 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.compaction_filter: None 2024-01-24T13:31:01.917 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.compaction_filter_factory: None 2024-01-24T13:31:01.917 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.sst_partitioner_factory: None 2024-01-24T13:31:01.917 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.memtable_factory: SkipListFactory 2024-01-24T13:31:01.917 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.table_factory: BlockBasedTable 2024-01-24T13:31:01.917 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5610f733b560) 2024-01-24T13:31:01.918 INFO:journalctl@ceph.mon.c.smithi187.stdout: cache_index_and_filter_blocks: 1 2024-01-24T13:31:01.918 INFO:journalctl@ceph.mon.c.smithi187.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-24T13:31:01.918 INFO:journalctl@ceph.mon.c.smithi187.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-24T13:31:01.918 INFO:journalctl@ceph.mon.c.smithi187.stdout: pin_top_level_index_and_filter: 1 2024-01-24T13:31:01.918 INFO:journalctl@ceph.mon.c.smithi187.stdout: index_type: 0 2024-01-24T13:31:01.918 INFO:journalctl@ceph.mon.c.smithi187.stdout: data_block_index_type: 0 2024-01-24T13:31:01.918 INFO:journalctl@ceph.mon.c.smithi187.stdout: index_shortening: 1 2024-01-24T13:31:01.918 INFO:journalctl@ceph.mon.c.smithi187.stdout: data_block_hash_table_util_ratio: 0.750000 2024-01-24T13:31:01.918 INFO:journalctl@ceph.mon.c.smithi187.stdout: checksum: 4 2024-01-24T13:31:01.918 INFO:journalctl@ceph.mon.c.smithi187.stdout: no_block_cache: 0 2024-01-24T13:31:01.918 INFO:journalctl@ceph.mon.c.smithi187.stdout: block_cache: 0x5610f664d1f0 2024-01-24T13:31:01.918 INFO:journalctl@ceph.mon.c.smithi187.stdout: block_cache_name: BinnedLRUCache 2024-01-24T13:31:01.918 INFO:journalctl@ceph.mon.c.smithi187.stdout: block_cache_options: 2024-01-24T13:31:01.918 INFO:journalctl@ceph.mon.c.smithi187.stdout: capacity : 536870912 2024-01-24T13:31:01.918 INFO:journalctl@ceph.mon.c.smithi187.stdout: num_shard_bits : 4 2024-01-24T13:31:01.919 INFO:journalctl@ceph.mon.c.smithi187.stdout: strict_capacity_limit : 0 2024-01-24T13:31:01.919 INFO:journalctl@ceph.mon.c.smithi187.stdout: high_pri_pool_ratio: 0.000 2024-01-24T13:31:01.919 INFO:journalctl@ceph.mon.c.smithi187.stdout: block_cache_compressed: (nil) 2024-01-24T13:31:01.919 INFO:journalctl@ceph.mon.c.smithi187.stdout: persistent_cache: (nil) 2024-01-24T13:31:01.919 INFO:journalctl@ceph.mon.c.smithi187.stdout: block_size: 4096 2024-01-24T13:31:01.919 INFO:journalctl@ceph.mon.c.smithi187.stdout: block_size_deviation: 10 2024-01-24T13:31:01.919 INFO:journalctl@ceph.mon.c.smithi187.stdout: block_restart_interval: 16 2024-01-24T13:31:01.919 INFO:journalctl@ceph.mon.c.smithi187.stdout: index_block_restart_interval: 1 2024-01-24T13:31:01.919 INFO:journalctl@ceph.mon.c.smithi187.stdout: metadata_block_size: 4096 2024-01-24T13:31:01.919 INFO:journalctl@ceph.mon.c.smithi187.stdout: partition_filters: 0 2024-01-24T13:31:01.919 INFO:journalctl@ceph.mon.c.smithi187.stdout: use_delta_encoding: 1 2024-01-24T13:31:01.919 INFO:journalctl@ceph.mon.c.smithi187.stdout: filter_policy: bloomfilter 2024-01-24T13:31:01.919 INFO:journalctl@ceph.mon.c.smithi187.stdout: whole_key_filtering: 1 2024-01-24T13:31:01.919 INFO:journalctl@ceph.mon.c.smithi187.stdout: verify_compression: 0 2024-01-24T13:31:01.919 INFO:journalctl@ceph.mon.c.smithi187.stdout: read_amp_bytes_per_bit: 0 2024-01-24T13:31:01.919 INFO:journalctl@ceph.mon.c.smithi187.stdout: format_version: 5 2024-01-24T13:31:01.920 INFO:journalctl@ceph.mon.c.smithi187.stdout: enable_index_compression: 1 2024-01-24T13:31:01.920 INFO:journalctl@ceph.mon.c.smithi187.stdout: block_align: 0 2024-01-24T13:31:01.920 INFO:journalctl@ceph.mon.c.smithi187.stdout: max_auto_readahead_size: 262144 2024-01-24T13:31:01.920 INFO:journalctl@ceph.mon.c.smithi187.stdout: prepopulate_block_cache: 0 2024-01-24T13:31:01.920 INFO:journalctl@ceph.mon.c.smithi187.stdout: initial_auto_readahead_size: 8192 2024-01-24T13:31:01.920 INFO:journalctl@ceph.mon.c.smithi187.stdout: num_file_reads_for_auto_readahead: 2 2024-01-24T13:31:01.920 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.write_buffer_size: 33554432 2024-01-24T13:31:01.920 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.max_write_buffer_number: 2 2024-01-24T13:31:01.920 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.compression: NoCompression 2024-01-24T13:31:01.920 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.bottommost_compression: Disabled 2024-01-24T13:31:01.920 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.prefix_extractor: nullptr 2024-01-24T13:31:01.920 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-24T13:31:01.920 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.num_levels: 7 2024-01-24T13:31:01.920 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-24T13:31:01.921 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-24T13:31:01.921 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-24T13:31:01.921 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-24T13:31:01.921 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-24T13:31:01.921 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-24T13:31:01.921 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-24T13:31:01.921 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-24T13:31:01.921 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-01-24T13:31:01.921 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-24T13:31:01.921 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-01-24T13:31:01.921 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-01-24T13:31:01.921 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.compression_opts.window_bits: -14 2024-01-24T13:31:01.921 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.compression_opts.level: 32767 2024-01-24T13:31:01.921 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.compression_opts.strategy: 0 2024-01-24T13:31:01.922 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-24T13:31:01.922 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-24T13:31:01.922 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-01-24T13:31:01.922 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-01-24T13:31:01.922 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.compression_opts.enabled: false 2024-01-24T13:31:01.922 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-01-24T13:31:01.922 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-24T13:31:01.922 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-24T13:31:01.922 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-24T13:31:01.922 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.target_file_size_base: 67108864 2024-01-24T13:31:01.922 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.target_file_size_multiplier: 1 2024-01-24T13:31:01.922 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-24T13:31:01.922 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-24T13:31:01.922 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-24T13:31:01.922 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-24T13:31:01.923 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-24T13:31:01.923 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-24T13:31:01.923 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-24T13:31:01.923 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-24T13:31:01.923 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-24T13:31:01.923 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-24T13:31:01.923 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-24T13:31:01.923 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-24T13:31:01.923 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-01-24T13:31:01.923 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.arena_block_size: 1048576 2024-01-24T13:31:01.923 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-24T13:31:01.923 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-24T13:31:01.923 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.disable_auto_compactions: 0 2024-01-24T13:31:01.923 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-24T13:31:01.923 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-24T13:31:01.923 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-24T13:31:01.924 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-24T13:31:01.925 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-24T13:31:01.925 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-24T13:31:01.925 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-24T13:31:01.925 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-24T13:31:01.925 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-24T13:31:01.925 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-24T13:31:01.925 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.table_properties_collectors: 2024-01-24T13:31:01.925 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.inplace_update_support: 0 2024-01-24T13:31:01.925 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.inplace_update_num_locks: 10000 2024-01-24T13:31:01.925 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-24T13:31:01.925 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-24T13:31:01.925 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.memtable_huge_page_size: 0 2024-01-24T13:31:01.925 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.bloom_locality: 0 2024-01-24T13:31:01.925 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.max_successive_merges: 0 2024-01-24T13:31:01.926 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.optimize_filters_for_hits: 0 2024-01-24T13:31:01.926 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.paranoid_file_checks: 0 2024-01-24T13:31:01.926 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.force_consistency_checks: 1 2024-01-24T13:31:01.926 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.report_bg_io_stats: 0 2024-01-24T13:31:01.926 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.ttl: 2592000 2024-01-24T13:31:01.926 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.periodic_compaction_seconds: 0 2024-01-24T13:31:01.926 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-01-24T13:31:01.926 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-01-24T13:31:01.926 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.enable_blob_files: false 2024-01-24T13:31:01.926 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.min_blob_size: 0 2024-01-24T13:31:01.926 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.blob_file_size: 268435456 2024-01-24T13:31:01.926 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.blob_compression_type: NoCompression 2024-01-24T13:31:01.926 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.enable_blob_garbage_collection: false 2024-01-24T13:31:01.926 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-01-24T13:31:01.926 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-01-24T13:31:01.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-01-24T13:31:01.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.blob_file_starting_level: 0 2024-01-24T13:31:01.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-01-24T13:31:01.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-24T13:31:01.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-01-24T13:31:01.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: ee6bbe41-bb16-4e3e-bdae-243736f8c726 2024-01-24T13:31:01.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: EVENT_LOG_v1 {"time_micros": 1706103061888106, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-01-24T13:31:01.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-01-24T13:31:01.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: EVENT_LOG_v1 {"time_micros": 1706103061888526, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1706103061, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ee6bbe41-bb16-4e3e-bdae-243736f8c726", "db_session_id": "LXA7L5U3VOR2WJO3VGUA", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-01-24T13:31:01.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: EVENT_LOG_v1 {"time_micros": 1706103061888590, "job": 1, "event": "recovery_finished"} 2024-01-24T13:31:01.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 bash[44860]: dfee0f0ff9c03ad5f13cafa9d59a934354957a9eb9598c84c2048e1a5bba5c5d 2024-01-24T13:31:01.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-01-24T13:31:01.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-01-24T13:31:01.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5610f66ea000 2024-01-24T13:31:01.928 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: DB pointer 0x5610f66d8000 2024-01-24T13:31:01.928 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-01-24T13:31:01.928 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-01-24T13:31:01.928 INFO:journalctl@ceph.mon.c.smithi187.stdout: ** DB Stats ** 2024-01-24T13:31:01.928 INFO:journalctl@ceph.mon.c.smithi187.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-01-24T13:31:01.928 INFO:journalctl@ceph.mon.c.smithi187.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-24T13:31:01.928 INFO:journalctl@ceph.mon.c.smithi187.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-24T13:31:01.928 INFO:journalctl@ceph.mon.c.smithi187.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-24T13:31:01.928 INFO:journalctl@ceph.mon.c.smithi187.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-24T13:31:01.928 INFO:journalctl@ceph.mon.c.smithi187.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-24T13:31:01.928 INFO:journalctl@ceph.mon.c.smithi187.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-24T13:31:01.928 INFO:journalctl@ceph.mon.c.smithi187.stdout: 2024-01-24T13:31:01.928 INFO:journalctl@ceph.mon.c.smithi187.stdout: ** Compaction Stats [default] ** 2024-01-24T13:31:01.928 INFO:journalctl@ceph.mon.c.smithi187.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-01-24T13:31:01.929 INFO:journalctl@ceph.mon.c.smithi187.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-01-24T13:31:01.929 INFO:journalctl@ceph.mon.c.smithi187.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-01-24T13:31:01.929 INFO:journalctl@ceph.mon.c.smithi187.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-01-24T13:31:01.929 INFO:journalctl@ceph.mon.c.smithi187.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-01-24T13:31:01.929 INFO:journalctl@ceph.mon.c.smithi187.stdout: 2024-01-24T13:31:01.929 INFO:journalctl@ceph.mon.c.smithi187.stdout: ** Compaction Stats [default] ** 2024-01-24T13:31:01.929 INFO:journalctl@ceph.mon.c.smithi187.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-01-24T13:31:01.929 INFO:journalctl@ceph.mon.c.smithi187.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-24T13:31:01.929 INFO:journalctl@ceph.mon.c.smithi187.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-01-24T13:31:01.929 INFO:journalctl@ceph.mon.c.smithi187.stdout: 2024-01-24T13:31:01.929 INFO:journalctl@ceph.mon.c.smithi187.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-01-24T13:31:01.929 INFO:journalctl@ceph.mon.c.smithi187.stdout: 2024-01-24T13:31:01.929 INFO:journalctl@ceph.mon.c.smithi187.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-01-24T13:31:01.930 INFO:journalctl@ceph.mon.c.smithi187.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-01-24T13:31:01.930 INFO:journalctl@ceph.mon.c.smithi187.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-24T13:31:01.930 INFO:journalctl@ceph.mon.c.smithi187.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-01-24T13:31:01.930 INFO:journalctl@ceph.mon.c.smithi187.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-01-24T13:31:01.930 INFO:journalctl@ceph.mon.c.smithi187.stdout: AddFile(Keys): cumulative 0, interval 0 2024-01-24T13:31:01.930 INFO:journalctl@ceph.mon.c.smithi187.stdout: Cumulative compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-24T13:31:01.930 INFO:journalctl@ceph.mon.c.smithi187.stdout: Interval compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-24T13:31:01.930 INFO:journalctl@ceph.mon.c.smithi187.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-24T13:31:01.930 INFO:journalctl@ceph.mon.c.smithi187.stdout: Block cache BinnedLRUCache@0x5610f664d1f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-01-24T13:31:01.930 INFO:journalctl@ceph.mon.c.smithi187.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-01-24T13:31:01.930 INFO:journalctl@ceph.mon.c.smithi187.stdout: 2024-01-24T13:31:01.930 INFO:journalctl@ceph.mon.c.smithi187.stdout: ** File Read Latency Histogram By Level [default] ** 2024-01-24T13:31:01.930 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-01-24T13:31:01.930 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: using public_addr v2:172.21.15.187:0/0 -> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] 2024-01-24T13:31:01.930 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: starting mon.c rank -1 at public addrs [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] at bind addrs [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 83afcf5e-babc-11ee-95b1-87774f69a715 2024-01-24T13:31:01.931 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: mon.c@-1(???) e0 preinit fsid 83afcf5e-babc-11ee-95b1-87774f69a715 2024-01-24T13:31:01.931 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 systemd[1]: Started Ceph mon.c for 83afcf5e-babc-11ee-95b1-87774f69a715. 2024-01-24T13:31:01.931 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: mon.c@-1(synchronizing).mds e1 new map 2024-01-24T13:31:01.931 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: mon.c@-1(synchronizing).mds e1 print_map 2024-01-24T13:31:01.931 INFO:journalctl@ceph.mon.c.smithi187.stdout: e1 2024-01-24T13:31:01.931 INFO:journalctl@ceph.mon.c.smithi187.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-01-24T13:31:01.931 INFO:journalctl@ceph.mon.c.smithi187.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-24T13:31:01.931 INFO:journalctl@ceph.mon.c.smithi187.stdout: legacy client fscid: -1 2024-01-24T13:31:01.931 INFO:journalctl@ceph.mon.c.smithi187.stdout: 2024-01-24T13:31:01.931 INFO:journalctl@ceph.mon.c.smithi187.stdout: No filesystems configured 2024-01-24T13:31:01.931 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-01-24T13:31:01.931 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-01-24T13:31:01.931 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-01-24T13:31:01.931 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-01-24T13:31:01.931 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-01-24T13:31:01.932 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-01-24T13:31:02.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-01-24T13:31:02.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-01-24T13:31:02.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-01-24T13:31:02.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-01-24T13:31:02.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-24T13:31:02.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:02.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:02.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:02.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:02.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:02.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi187", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:31:02.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:02.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:31:02.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: Updating smithi187:/etc/ceph/ceph.conf 2024-01-24T13:31:02.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/3350081638' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-01-24T13:31:02.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:02.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:02.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:02.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: Updating smithi187:/var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-24T13:31:02.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: Updating smithi187:/etc/ceph/ceph.client.admin.keyring 2024-01-24T13:31:02.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: Updating smithi187:/var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/config/ceph.client.admin.keyring 2024-01-24T13:31:02.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:02.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/3350081638' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-01-24T13:31:02.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: osdmap e4: 0 total, 0 up, 0 in 2024-01-24T13:31:02.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:02.307 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi029:172.21.15.29=a;smithi163:172.21.15.163=b;smithi187:172.21.15.187=c", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:31:02.307 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: Saving service mon spec with placement smithi029:172.21.15.29=a;smithi163:172.21.15.163=b;smithi187:172.21.15.187=c;count:3 2024-01-24T13:31:02.307 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:02.307 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:31:02.307 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:02.307 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:31:02.307 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:02.307 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-24T13:31:02.307 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:02.307 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: Deploying daemon mon.c on smithi187 2024-01-24T13:31:02.307 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-01-24T13:31:02.307 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2024-01-24T13:31:02.307 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: expand_channel_meta expand map: {default=false} 2024-01-24T13:31:02.307 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: expand_channel_meta from 'false' to 'false' 2024-01-24T13:31:02.307 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: expand_channel_meta expanded map: {default=false} 2024-01-24T13:31:02.307 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: expand_channel_meta expand map: {default=info} 2024-01-24T13:31:02.308 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: expand_channel_meta from 'info' to 'info' 2024-01-24T13:31:02.308 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: expand_channel_meta expanded map: {default=info} 2024-01-24T13:31:02.308 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: expand_channel_meta expand map: {default=daemon} 2024-01-24T13:31:02.308 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: expand_channel_meta from 'daemon' to 'daemon' 2024-01-24T13:31:02.308 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: expand_channel_meta expanded map: {default=daemon} 2024-01-24T13:31:02.308 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: expand_channel_meta expand map: {default=debug} 2024-01-24T13:31:02.308 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: expand_channel_meta from 'debug' to 'debug' 2024-01-24T13:31:02.308 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:01 smithi187 ceph-mon[44911]: expand_channel_meta expanded map: {default=debug} 2024-01-24T13:31:03.117 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-01-24T13:31:03.117 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph mon dump -f json 2024-01-24T13:31:03.348 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.c/config 2024-01-24T13:31:07.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:07 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-01-24T13:31:07.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:07 smithi187 ceph-mon[44911]: mon.a calling monitor election 2024-01-24T13:31:07.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:07 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T13:31:07.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:07 smithi187 ceph-mon[44911]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:07.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:07 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T13:31:07.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:07 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T13:31:07.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:07 smithi187 ceph-mon[44911]: mon.c calling monitor election 2024-01-24T13:31:07.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:07 smithi187 ceph-mon[44911]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:07.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:07 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T13:31:07.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:07 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T13:31:07.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:07 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T13:31:07.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:07 smithi187 ceph-mon[44911]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-01-24T13:31:07.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:07 smithi187 ceph-mon[44911]: monmap e2: 2 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],c=[v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-01-24T13:31:07.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:07 smithi187 ceph-mon[44911]: fsmap 2024-01-24T13:31:07.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:07 smithi187 ceph-mon[44911]: osdmap e4: 0 total, 0 up, 0 in 2024-01-24T13:31:07.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:07 smithi187 ceph-mon[44911]: mgrmap e13: x(active, since 36s) 2024-01-24T13:31:07.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:07 smithi187 ceph-mon[44911]: overall HEALTH_OK 2024-01-24T13:31:07.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:07 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:07.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:07 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:07.307 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:07 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:07.307 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:07 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-24T13:31:07.307 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:07 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:07.341 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.340+0000 7f773ca7f700 1 -- 172.21.15.187:0/2649530063 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77240035c0 msgr2=0x7f7724005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:31:07.341 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.340+0000 7f773ca7f700 1 --2- 172.21.15.187:0/2649530063 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77240035c0 0x7f7724005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7738075e40 tx=0x7f772000feb0 comp rx=0 tx=0).stop 2024-01-24T13:31:07.342 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.341+0000 7f773ca7f700 1 -- 172.21.15.187:0/2649530063 shutdown_connections 2024-01-24T13:31:07.342 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.341+0000 7f773ca7f700 1 --2- 172.21.15.187:0/2649530063 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77240035c0 0x7f7724005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:07.342 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.341+0000 7f773ca7f700 1 -- 172.21.15.187:0/2649530063 >> 172.21.15.187:0/2649530063 conn(0x7f7738074a90 msgr2=0x7f7738074ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:31:07.342 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.341+0000 7f773ca7f700 1 -- 172.21.15.187:0/2649530063 shutdown_connections 2024-01-24T13:31:07.342 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.341+0000 7f773ca7f700 1 -- 172.21.15.187:0/2649530063 wait complete. 2024-01-24T13:31:07.343 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.342+0000 7f773ca7f700 1 Processor -- start 2024-01-24T13:31:07.343 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.342+0000 7f773ca7f700 1 -- start start 2024-01-24T13:31:07.344 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.342+0000 7f773ca7f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77240035c0 0x7f7738195f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:31:07.344 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.342+0000 7f773ca7f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77381964b0 0x7f773819a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:31:07.344 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.342+0000 7f773ca7f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7738196a40 con 0x7f77240035c0 2024-01-24T13:31:07.344 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.342+0000 7f773ca7f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7738196bb0 con 0x7f77381964b0 2024-01-24T13:31:07.344 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.342+0000 7f7735d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77381964b0 0x7f773819a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:31:07.344 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.342+0000 7f7735d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77381964b0 0x7f773819a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:56550/0 (socket says 172.21.15.187:56550) 2024-01-24T13:31:07.344 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.342+0000 7f7735d9b700 1 -- 172.21.15.187:0/1399570328 learned_addr learned my addr 172.21.15.187:0/1399570328 (peer_addr_for_me v2:172.21.15.187:0/0) 2024-01-24T13:31:07.344 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.342+0000 7f773659c700 1 --2- 172.21.15.187:0/1399570328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77240035c0 0x7f7738195f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:31:07.344 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.342+0000 7f7735d9b700 1 -- 172.21.15.187:0/1399570328 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77381964b0 msgr2=0x7f773819a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-01-24T13:31:07.344 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.342+0000 7f7735d9b700 1 -- 172.21.15.187:0/1399570328 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77381964b0 msgr2=0x7f773819a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-01-24T13:31:07.344 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.342+0000 7f7735d9b700 1 --2- 172.21.15.187:0/1399570328 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77381964b0 0x7f773819a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-24T13:31:07.344 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.342+0000 7f7735d9b700 1 --2- 172.21.15.187:0/1399570328 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77381964b0 0x7f773819a920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-24T13:31:07.344 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.343+0000 7f773659c700 1 -- 172.21.15.187:0/1399570328 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77381964b0 msgr2=0x7f773819a920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-24T13:31:07.345 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.343+0000 7f773659c700 1 --2- 172.21.15.187:0/1399570328 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77381964b0 0x7f773819a920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:07.345 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.343+0000 7f773659c700 1 -- 172.21.15.187:0/1399570328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f772000f900 con 0x7f77240035c0 2024-01-24T13:31:07.345 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.343+0000 7f773659c700 1 --2- 172.21.15.187:0/1399570328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77240035c0 0x7f7738195f70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7720000c00 tx=0x7f77200047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:31:07.346 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.343+0000 7f772f7fe700 1 -- 172.21.15.187:0/1399570328 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f7720013b00 con 0x7f77240035c0 2024-01-24T13:31:07.346 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.343+0000 7f773ca7f700 1 -- 172.21.15.187:0/1399570328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f773819aec0 con 0x7f77240035c0 2024-01-24T13:31:07.346 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.343+0000 7f772f7fe700 1 -- 172.21.15.187:0/1399570328 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f77200044e0 con 0x7f77240035c0 2024-01-24T13:31:07.346 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.343+0000 7f772f7fe700 1 -- 172.21.15.187:0/1399570328 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f7720003c20 con 0x7f77240035c0 2024-01-24T13:31:07.346 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.343+0000 7f773ca7f700 1 -- 172.21.15.187:0/1399570328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f773819b410 con 0x7f77240035c0 2024-01-24T13:31:07.346 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.344+0000 7f772f7fe700 1 -- 172.21.15.187:0/1399570328 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45324+0+0 (secure 0 0 0) 0x7f7720011590 con 0x7f77240035c0 2024-01-24T13:31:07.346 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.344+0000 7f772f7fe700 1 --2- 172.21.15.187:0/1399570328 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f7718038180 0x7f771803a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:31:07.346 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.344+0000 7f773ca7f700 1 -- 172.21.15.187:0/1399570328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7738062370 con 0x7f77240035c0 2024-01-24T13:31:07.347 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.344+0000 7f772f7fe700 1 -- 172.21.15.187:0/1399570328 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f772004db60 con 0x7f77240035c0 2024-01-24T13:31:07.347 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.345+0000 7f7735d9b700 1 --2- 172.21.15.187:0/1399570328 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f7718038180 0x7f771803a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:31:07.347 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.345+0000 7f7735d9b700 1 --2- 172.21.15.187:0/1399570328 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f7718038180 0x7f771803a640 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7728009180 tx=0x7f7728008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:31:07.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:07 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-01-24T13:31:07.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:07 smithi029 ceph-mon[40651]: mon.a calling monitor election 2024-01-24T13:31:07.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:07 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T13:31:07.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:07 smithi029 ceph-mon[40651]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:07.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:07 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T13:31:07.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:07 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T13:31:07.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:07 smithi029 ceph-mon[40651]: mon.c calling monitor election 2024-01-24T13:31:07.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:07 smithi029 ceph-mon[40651]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:07.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:07 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T13:31:07.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:07 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T13:31:07.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:07 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T13:31:07.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:07 smithi029 ceph-mon[40651]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-01-24T13:31:07.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:07 smithi029 ceph-mon[40651]: monmap e2: 2 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],c=[v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-01-24T13:31:07.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:07 smithi029 ceph-mon[40651]: fsmap 2024-01-24T13:31:07.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:07 smithi029 ceph-mon[40651]: osdmap e4: 0 total, 0 up, 0 in 2024-01-24T13:31:07.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:07 smithi029 ceph-mon[40651]: mgrmap e13: x(active, since 36s) 2024-01-24T13:31:07.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:07 smithi029 ceph-mon[40651]: overall HEALTH_OK 2024-01-24T13:31:07.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:07 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:07.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:07 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:07.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:07 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:07.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:07 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-24T13:31:07.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:07 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:07.350 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.350+0000 7f772f7fe700 1 -- 172.21.15.187:0/1399570328 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f7720024400 con 0x7f77240035c0 2024-01-24T13:31:07.618 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.617+0000 7f773ca7f700 1 -- 172.21.15.187:0/1399570328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f773819b6e0 con 0x7f77240035c0 2024-01-24T13:31:07.618 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.617+0000 7f772f7fe700 1 -- 172.21.15.187:0/1399570328 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1006 (secure 0 0 0) 0x7f772002bb90 con 0x7f77240035c0 2024-01-24T13:31:07.619 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-01-24T13:31:07.619 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":2,"fsid":"83afcf5e-babc-11ee-95b1-87774f69a715","modified":"2024-01-24T13:31:01.917792Z","created":"2024-01-24T13:29:31.023487Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:3300","nonce":0},{"type":"v1","addr":"172.21.15.187:6789","nonce":0}]},"addr":"172.21.15.187:6789/0","public_addr":"172.21.15.187:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-01-24T13:31:07.621 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.619+0000 7f773ca7f700 1 -- 172.21.15.187:0/1399570328 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f7718038180 msgr2=0x7f771803a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:31:07.621 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.619+0000 7f773ca7f700 1 --2- 172.21.15.187:0/1399570328 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f7718038180 0x7f771803a640 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7728009180 tx=0x7f7728008040 comp rx=0 tx=0).stop 2024-01-24T13:31:07.621 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.619+0000 7f773ca7f700 1 -- 172.21.15.187:0/1399570328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77240035c0 msgr2=0x7f7738195f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:31:07.621 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.619+0000 7f773ca7f700 1 --2- 172.21.15.187:0/1399570328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77240035c0 0x7f7738195f70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7720000c00 tx=0x7f77200047a0 comp rx=0 tx=0).stop 2024-01-24T13:31:07.621 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.619+0000 7f773ca7f700 1 -- 172.21.15.187:0/1399570328 shutdown_connections 2024-01-24T13:31:07.621 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.619+0000 7f773ca7f700 1 --2- 172.21.15.187:0/1399570328 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f7718038180 0x7f771803a640 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:07.621 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.619+0000 7f773ca7f700 1 --2- 172.21.15.187:0/1399570328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77240035c0 0x7f7738195f70 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:07.621 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.619+0000 7f773ca7f700 1 --2- 172.21.15.187:0/1399570328 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77381964b0 0x7f773819a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:07.621 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.620+0000 7f773ca7f700 1 -- 172.21.15.187:0/1399570328 >> 172.21.15.187:0/1399570328 conn(0x7f7738074a90 msgr2=0x7f773818dd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:31:07.621 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.620+0000 7f773ca7f700 1 -- 172.21.15.187:0/1399570328 shutdown_connections 2024-01-24T13:31:07.621 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:07.620+0000 7f773ca7f700 1 -- 172.21.15.187:0/1399570328 wait complete. 2024-01-24T13:31:07.621 INFO:teuthology.orchestra.run.smithi187.stderr:dumped monmap epoch 2 2024-01-24T13:31:08.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:08 smithi187 ceph-mon[44911]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:08.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:08 smithi187 ceph-mon[44911]: Deploying daemon mon.b on smithi163 2024-01-24T13:31:08.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:08 smithi187 ceph-mon[44911]: from='client.? 172.21.15.187:0/1399570328' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-24T13:31:08.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:08 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T13:31:08.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:08 smithi029 ceph-mon[40651]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:08.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:08 smithi029 ceph-mon[40651]: Deploying daemon mon.b on smithi163 2024-01-24T13:31:08.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:08 smithi029 ceph-mon[40651]: from='client.? 172.21.15.187:0/1399570328' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-24T13:31:08.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:08 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T13:31:09.007 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-01-24T13:31:09.008 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph mon dump -f json 2024-01-24T13:31:09.237 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.c/config 2024-01-24T13:31:09.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 systemd[1]: Started Ceph mon.b for 83afcf5e-babc-11ee-95b1-87774f69a715. 2024-01-24T13:31:09.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: mon.b@-1(synchronizing).mds e1 new map 2024-01-24T13:31:09.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: mon.b@-1(synchronizing).mds e1 print_map 2024-01-24T13:31:09.983 INFO:journalctl@ceph.mon.b.smithi163.stdout: e1 2024-01-24T13:31:09.983 INFO:journalctl@ceph.mon.b.smithi163.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-01-24T13:31:09.983 INFO:journalctl@ceph.mon.b.smithi163.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-24T13:31:09.983 INFO:journalctl@ceph.mon.b.smithi163.stdout: legacy client fscid: -1 2024-01-24T13:31:09.983 INFO:journalctl@ceph.mon.b.smithi163.stdout: 2024-01-24T13:31:09.983 INFO:journalctl@ceph.mon.b.smithi163.stdout: No filesystems configured 2024-01-24T13:31:09.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-01-24T13:31:09.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-01-24T13:31:09.984 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-01-24T13:31:09.984 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-01-24T13:31:09.984 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-01-24T13:31:09.984 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-01-24T13:31:09.984 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-01-24T13:31:09.984 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-01-24T13:31:09.984 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-01-24T13:31:09.984 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-01-24T13:31:09.984 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-24T13:31:09.984 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:09.984 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:09.984 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:09.984 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:09.984 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:09.985 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi187", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:31:09.985 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:09.985 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:31:09.985 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: Updating smithi187:/etc/ceph/ceph.conf 2024-01-24T13:31:09.985 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/3350081638' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-01-24T13:31:09.985 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:09.985 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:09.985 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:09.985 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: Updating smithi187:/var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-24T13:31:09.985 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: Updating smithi187:/etc/ceph/ceph.client.admin.keyring 2024-01-24T13:31:09.985 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: Updating smithi187:/var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/config/ceph.client.admin.keyring 2024-01-24T13:31:09.985 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:09.985 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/3350081638' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-01-24T13:31:09.985 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: osdmap e4: 0 total, 0 up, 0 in 2024-01-24T13:31:09.985 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:09.986 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi029:172.21.15.29=a;smithi163:172.21.15.163=b;smithi187:172.21.15.187=c", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:31:09.986 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: Saving service mon spec with placement smithi029:172.21.15.29=a;smithi163:172.21.15.163=b;smithi187:172.21.15.187=c;count:3 2024-01-24T13:31:09.986 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:09.986 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:31:09.986 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:09.986 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:31:09.986 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:09.986 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-24T13:31:09.986 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:09.986 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: Deploying daemon mon.c on smithi187 2024-01-24T13:31:09.986 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-01-24T13:31:09.986 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: mon.a calling monitor election 2024-01-24T13:31:09.986 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T13:31:09.986 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:09.987 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T13:31:09.987 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T13:31:09.987 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: mon.c calling monitor election 2024-01-24T13:31:09.987 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:09.987 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T13:31:09.987 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T13:31:09.987 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T13:31:09.987 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-01-24T13:31:09.987 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: monmap e2: 2 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],c=[v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-01-24T13:31:09.987 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: fsmap 2024-01-24T13:31:09.987 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: osdmap e4: 0 total, 0 up, 0 in 2024-01-24T13:31:09.987 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: mgrmap e13: x(active, since 36s) 2024-01-24T13:31:09.987 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: overall HEALTH_OK 2024-01-24T13:31:09.987 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:09.987 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:09.988 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:09.988 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-24T13:31:09.988 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:09.988 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:09.988 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: Deploying daemon mon.b on smithi163 2024-01-24T13:31:09.988 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: from='client.? 172.21.15.187:0/1399570328' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-24T13:31:09.988 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T13:31:09.988 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-01-24T13:31:09.988 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete 2024-01-24T13:31:09.988 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: expand_channel_meta expand map: {default=false} 2024-01-24T13:31:09.988 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: expand_channel_meta from 'false' to 'false' 2024-01-24T13:31:09.988 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: expand_channel_meta expanded map: {default=false} 2024-01-24T13:31:09.988 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: expand_channel_meta expand map: {default=info} 2024-01-24T13:31:09.988 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: expand_channel_meta from 'info' to 'info' 2024-01-24T13:31:09.989 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: expand_channel_meta expanded map: {default=info} 2024-01-24T13:31:09.989 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: expand_channel_meta expand map: {default=daemon} 2024-01-24T13:31:09.989 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: expand_channel_meta from 'daemon' to 'daemon' 2024-01-24T13:31:09.989 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: expand_channel_meta expanded map: {default=daemon} 2024-01-24T13:31:09.989 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: expand_channel_meta expand map: {default=debug} 2024-01-24T13:31:09.989 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: expand_channel_meta from 'debug' to 'debug' 2024-01-24T13:31:09.989 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:09 smithi163 ceph-mon[44969]: expand_channel_meta expanded map: {default=debug} 2024-01-24T13:31:14.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:14 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-01-24T13:31:14.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:14 smithi163 ceph-mon[44969]: mon.a calling monitor election 2024-01-24T13:31:14.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:14 smithi163 ceph-mon[44969]: mon.c calling monitor election 2024-01-24T13:31:14.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:14 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T13:31:14.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:14 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T13:31:14.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:14 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T13:31:14.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:14 smithi163 ceph-mon[44969]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:14.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:14 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T13:31:14.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:14 smithi163 ceph-mon[44969]: mon.b calling monitor election 2024-01-24T13:31:14.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:14 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T13:31:14.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:14 smithi163 ceph-mon[44969]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:14.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:14 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T13:31:14.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:14 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T13:31:14.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:14 smithi163 ceph-mon[44969]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-01-24T13:31:14.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:14 smithi163 ceph-mon[44969]: monmap e3: 3 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],b=[v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0],c=[v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-01-24T13:31:14.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:14 smithi163 ceph-mon[44969]: fsmap 2024-01-24T13:31:14.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:14 smithi163 ceph-mon[44969]: osdmap e4: 0 total, 0 up, 0 in 2024-01-24T13:31:14.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:14 smithi163 ceph-mon[44969]: mgrmap e13: x(active, since 43s) 2024-01-24T13:31:14.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:14 smithi163 ceph-mon[44969]: overall HEALTH_OK 2024-01-24T13:31:14.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:14 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:14.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:14 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:14.984 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:14 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:15.054 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:14 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-01-24T13:31:15.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:14 smithi187 ceph-mon[44911]: mon.a calling monitor election 2024-01-24T13:31:15.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:14 smithi187 ceph-mon[44911]: mon.c calling monitor election 2024-01-24T13:31:15.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:14 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T13:31:15.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:14 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T13:31:15.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:14 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T13:31:15.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:14 smithi187 ceph-mon[44911]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:15.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:14 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T13:31:15.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:14 smithi187 ceph-mon[44911]: mon.b calling monitor election 2024-01-24T13:31:15.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:14 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T13:31:15.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:14 smithi187 ceph-mon[44911]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:15.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:14 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T13:31:15.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:14 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T13:31:15.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:14 smithi187 ceph-mon[44911]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-01-24T13:31:15.056 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:14 smithi187 ceph-mon[44911]: monmap e3: 3 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],b=[v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0],c=[v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-01-24T13:31:15.056 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:14 smithi187 ceph-mon[44911]: fsmap 2024-01-24T13:31:15.056 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:14 smithi187 ceph-mon[44911]: osdmap e4: 0 total, 0 up, 0 in 2024-01-24T13:31:15.056 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:14 smithi187 ceph-mon[44911]: mgrmap e13: x(active, since 43s) 2024-01-24T13:31:15.056 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:14 smithi187 ceph-mon[44911]: overall HEALTH_OK 2024-01-24T13:31:15.056 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:14 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:15.056 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:14 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:15.056 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:14 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:15.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:14 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-01-24T13:31:15.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:14 smithi029 ceph-mon[40651]: mon.a calling monitor election 2024-01-24T13:31:15.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:14 smithi029 ceph-mon[40651]: mon.c calling monitor election 2024-01-24T13:31:15.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:14 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T13:31:15.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:14 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T13:31:15.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:14 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T13:31:15.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:14 smithi029 ceph-mon[40651]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:15.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:14 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T13:31:15.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:14 smithi029 ceph-mon[40651]: mon.b calling monitor election 2024-01-24T13:31:15.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:14 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T13:31:15.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:14 smithi029 ceph-mon[40651]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:15.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:14 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T13:31:15.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:14 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T13:31:15.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:14 smithi029 ceph-mon[40651]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-01-24T13:31:15.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:14 smithi029 ceph-mon[40651]: monmap e3: 3 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],b=[v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0],c=[v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-01-24T13:31:15.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:14 smithi029 ceph-mon[40651]: fsmap 2024-01-24T13:31:15.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:14 smithi029 ceph-mon[40651]: osdmap e4: 0 total, 0 up, 0 in 2024-01-24T13:31:15.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:14 smithi029 ceph-mon[40651]: mgrmap e13: x(active, since 43s) 2024-01-24T13:31:15.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:14 smithi029 ceph-mon[40651]: overall HEALTH_OK 2024-01-24T13:31:15.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:14 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:15.101 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:14 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:15.101 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:14 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:15.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:15 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:15.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:15 smithi163 ceph-mon[44969]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:15.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:15 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:31:15.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:15 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:15.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:15 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:15.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:15 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T13:31:15.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:15 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:15.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:15 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:15.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:15 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:15.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:15 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:31:16.054 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:15 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:16.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:15 smithi187 ceph-mon[44911]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:16.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:15 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:31:16.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:15 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:16.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:15 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:16.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:15 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T13:31:16.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:15 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:16.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:15 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:16.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:15 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:16.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:15 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:31:16.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:15 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:16.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:15 smithi029 ceph-mon[40651]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:16.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:15 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:31:16.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:15 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:16.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:15 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:16.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:15 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T13:31:16.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:15 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:16.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:15 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:16.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:15 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:16.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:15 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:31:16.152 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.151+0000 7f43883c3700 1 -- 172.21.15.187:0/1623922134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f436c006230 msgr2=0x7f436c0039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:31:16.152 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.151+0000 7f43883c3700 1 --2- 172.21.15.187:0/1623922134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f436c006230 0x7f436c0039c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f43801008b0 tx=0x7f437c00fe90 comp rx=0 tx=0).stop 2024-01-24T13:31:16.152 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.151+0000 7f43883c3700 1 -- 172.21.15.187:0/1623922134 shutdown_connections 2024-01-24T13:31:16.152 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.151+0000 7f43883c3700 1 --2- 172.21.15.187:0/1623922134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f436c006230 0x7f436c0039c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:16.152 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.151+0000 7f43883c3700 1 -- 172.21.15.187:0/1623922134 >> 172.21.15.187:0/1623922134 conn(0x7f43800752f0 msgr2=0x7f4380075700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:31:16.152 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.151+0000 7f43883c3700 1 -- 172.21.15.187:0/1623922134 shutdown_connections 2024-01-24T13:31:16.152 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.151+0000 7f43883c3700 1 -- 172.21.15.187:0/1623922134 wait complete. 2024-01-24T13:31:16.155 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.152+0000 7f43883c3700 1 Processor -- start 2024-01-24T13:31:16.155 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.152+0000 7f43883c3700 1 -- start start 2024-01-24T13:31:16.155 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.152+0000 7f43883c3700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f436c006230 0x7f4380194f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:31:16.155 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.152+0000 7f43883c3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43801954b0 0x7f4380199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:31:16.155 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.152+0000 7f43883c3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4380199e60 0x7f438019a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:31:16.155 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.152+0000 7f43883c3700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4380195ba0 con 0x7f4380199e60 2024-01-24T13:31:16.155 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.152+0000 7f43883c3700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43801a1bd0 con 0x7f436c006230 2024-01-24T13:31:16.155 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.152+0000 7f43883c3700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43801a1d10 con 0x7f43801954b0 2024-01-24T13:31:16.156 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.152+0000 7f438595e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43801954b0 0x7f4380199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:31:16.156 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.152+0000 7f438595e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43801954b0 0x7f4380199920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48226/0 (socket says 172.21.15.187:48226) 2024-01-24T13:31:16.156 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.152+0000 7f438595e700 1 -- 172.21.15.187:0/2081034883 learned_addr learned my addr 172.21.15.187:0/2081034883 (peer_addr_for_me v2:172.21.15.187:0/0) 2024-01-24T13:31:16.156 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.152+0000 7f438615f700 1 --2- 172.21.15.187:0/2081034883 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f436c006230 0x7f4380194f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:31:16.156 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.152+0000 7f4386960700 1 --2- 172.21.15.187:0/2081034883 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4380199e60 0x7f438019a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:31:16.156 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.152+0000 7f438595e700 1 -- 172.21.15.187:0/2081034883 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43801954b0 msgr2=0x7f4380199920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-01-24T13:31:16.156 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.152+0000 7f438595e700 1 -- 172.21.15.187:0/2081034883 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43801954b0 msgr2=0x7f4380199920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-01-24T13:31:16.156 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.152+0000 7f438595e700 1 --2- 172.21.15.187:0/2081034883 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43801954b0 0x7f4380199920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-24T13:31:16.156 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.152+0000 7f438595e700 1 --2- 172.21.15.187:0/2081034883 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43801954b0 0x7f4380199920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-24T13:31:16.156 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.152+0000 7f438615f700 1 -- 172.21.15.187:0/2081034883 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f436c006230 msgr2=0x7f4380194f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-01-24T13:31:16.156 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.152+0000 7f438615f700 1 -- 172.21.15.187:0/2081034883 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f436c006230 msgr2=0x7f4380194f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-01-24T13:31:16.156 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.152+0000 7f438615f700 1 --2- 172.21.15.187:0/2081034883 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f436c006230 0x7f4380194f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-24T13:31:16.156 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.152+0000 7f438615f700 1 --2- 172.21.15.187:0/2081034883 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f436c006230 0x7f4380194f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-24T13:31:16.156 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.153+0000 7f4386960700 1 -- 172.21.15.187:0/2081034883 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43801954b0 msgr2=0x7f4380199920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-24T13:31:16.157 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.153+0000 7f4386960700 1 --2- 172.21.15.187:0/2081034883 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43801954b0 0x7f4380199920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:16.157 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.153+0000 7f4386960700 1 -- 172.21.15.187:0/2081034883 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f436c006230 msgr2=0x7f4380194f70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-24T13:31:16.157 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.153+0000 7f4386960700 1 --2- 172.21.15.187:0/2081034883 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f436c006230 0x7f4380194f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:16.157 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.153+0000 7f4386960700 1 -- 172.21.15.187:0/2081034883 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f437c00f8e0 con 0x7f4380199e60 2024-01-24T13:31:16.157 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.153+0000 7f4386960700 1 --2- 172.21.15.187:0/2081034883 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4380199e60 0x7f438019a310 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f437800bd50 tx=0x7f437800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:31:16.157 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.153+0000 7f43777fe700 1 -- 172.21.15.187:0/2081034883 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f437800cd10 con 0x7f4380199e60 2024-01-24T13:31:16.157 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.153+0000 7f43883c3700 1 -- 172.21.15.187:0/2081034883 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4380068500 con 0x7f4380199e60 2024-01-24T13:31:16.157 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.153+0000 7f43777fe700 1 -- 172.21.15.187:0/2081034883 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f437800ce70 con 0x7f4380199e60 2024-01-24T13:31:16.157 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.153+0000 7f43777fe700 1 -- 172.21.15.187:0/2081034883 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4378012840 con 0x7f4380199e60 2024-01-24T13:31:16.157 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.153+0000 7f43883c3700 1 -- 172.21.15.187:0/2081034883 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43800689b0 con 0x7f4380199e60 2024-01-24T13:31:16.157 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.154+0000 7f43777fe700 1 -- 172.21.15.187:0/2081034883 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45324+0+0 (secure 0 0 0) 0x7f4378014440 con 0x7f4380199e60 2024-01-24T13:31:16.157 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.154+0000 7f43777fe700 1 --2- 172.21.15.187:0/2081034883 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f4364038820 0x7f436403ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:31:16.157 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.154+0000 7f43777fe700 1 -- 172.21.15.187:0/2081034883 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f437804a8c0 con 0x7f4380199e60 2024-01-24T13:31:16.157 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.154+0000 7f43883c3700 1 -- 172.21.15.187:0/2081034883 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4380069400 con 0x7f4380199e60 2024-01-24T13:31:16.158 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.154+0000 7f438615f700 1 --2- 172.21.15.187:0/2081034883 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f4364038820 0x7f436403ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:31:16.158 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.155+0000 7f438615f700 1 --2- 172.21.15.187:0/2081034883 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f4364038820 0x7f436403ace0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f437c010010 tx=0x7f437c001e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:31:16.160 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.160+0000 7f43777fe700 1 -- 172.21.15.187:0/2081034883 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f4378014b00 con 0x7f4380199e60 2024-01-24T13:31:16.430 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.429+0000 7f43883c3700 1 -- 172.21.15.187:0/2081034883 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f43801962d0 con 0x7f4380199e60 2024-01-24T13:31:16.430 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.429+0000 7f43777fe700 1 -- 172.21.15.187:0/2081034883 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1276 (secure 0 0 0) 0x7f437801b8d0 con 0x7f4380199e60 2024-01-24T13:31:16.430 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-01-24T13:31:16.430 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":3,"fsid":"83afcf5e-babc-11ee-95b1-87774f69a715","modified":"2024-01-24T13:31:09.590504Z","created":"2024-01-24T13:29:31.023487Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:3300","nonce":0},{"type":"v1","addr":"172.21.15.187:6789","nonce":0}]},"addr":"172.21.15.187:6789/0","public_addr":"172.21.15.187:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-01-24T13:31:16.432 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.432+0000 7f43883c3700 1 -- 172.21.15.187:0/2081034883 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f4364038820 msgr2=0x7f436403ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:31:16.432 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.432+0000 7f43883c3700 1 --2- 172.21.15.187:0/2081034883 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f4364038820 0x7f436403ace0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f437c010010 tx=0x7f437c001e50 comp rx=0 tx=0).stop 2024-01-24T13:31:16.433 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.432+0000 7f43883c3700 1 -- 172.21.15.187:0/2081034883 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4380199e60 msgr2=0x7f438019a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:31:16.433 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.432+0000 7f43883c3700 1 --2- 172.21.15.187:0/2081034883 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4380199e60 0x7f438019a310 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f437800bd50 tx=0x7f437800be30 comp rx=0 tx=0).stop 2024-01-24T13:31:16.433 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.432+0000 7f43883c3700 1 -- 172.21.15.187:0/2081034883 shutdown_connections 2024-01-24T13:31:16.433 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.432+0000 7f43883c3700 1 --2- 172.21.15.187:0/2081034883 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f4364038820 0x7f436403ace0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:16.433 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.432+0000 7f43883c3700 1 --2- 172.21.15.187:0/2081034883 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4380199e60 0x7f438019a310 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:16.433 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.432+0000 7f43883c3700 1 --2- 172.21.15.187:0/2081034883 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f436c006230 0x7f4380194f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:16.433 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.432+0000 7f43883c3700 1 --2- 172.21.15.187:0/2081034883 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43801954b0 0x7f4380199920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:16.433 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.432+0000 7f43883c3700 1 -- 172.21.15.187:0/2081034883 >> 172.21.15.187:0/2081034883 conn(0x7f43800752f0 msgr2=0x7f43801014d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:31:16.433 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.432+0000 7f43883c3700 1 -- 172.21.15.187:0/2081034883 shutdown_connections 2024-01-24T13:31:16.433 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:16.432+0000 7f43883c3700 1 -- 172.21.15.187:0/2081034883 wait complete. 2024-01-24T13:31:16.433 INFO:teuthology.orchestra.run.smithi187.stderr:dumped monmap epoch 3 2024-01-24T13:31:16.753 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-01-24T13:31:16.754 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph config generate-minimal-conf 2024-01-24T13:31:17.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:16 smithi187 ceph-mon[44911]: Updating smithi029:/etc/ceph/ceph.conf 2024-01-24T13:31:17.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:16 smithi187 ceph-mon[44911]: Updating smithi163:/etc/ceph/ceph.conf 2024-01-24T13:31:17.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:16 smithi187 ceph-mon[44911]: Updating smithi187:/etc/ceph/ceph.conf 2024-01-24T13:31:17.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:16 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:17.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:16 smithi187 ceph-mon[44911]: Updating smithi163:/var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-24T13:31:17.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:16 smithi187 ceph-mon[44911]: Updating smithi187:/var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-24T13:31:17.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:16 smithi187 ceph-mon[44911]: Updating smithi029:/var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-24T13:31:17.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:16 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:17.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:16 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:17.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:16 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:17.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:16 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:17.056 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:16 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:17.056 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:16 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:17.056 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:16 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:17.056 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:16 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:17.056 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:16 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:17.056 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:16 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:17.056 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:16 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:17.056 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:16 smithi187 ceph-mon[44911]: from='client.? 172.21.15.187:0/2081034883' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-24T13:31:17.056 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:16 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-24T13:31:17.056 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:16 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-24T13:31:17.056 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:16 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:17.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:16 smithi029 ceph-mon[40651]: Updating smithi029:/etc/ceph/ceph.conf 2024-01-24T13:31:17.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:16 smithi029 ceph-mon[40651]: Updating smithi163:/etc/ceph/ceph.conf 2024-01-24T13:31:17.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:16 smithi029 ceph-mon[40651]: Updating smithi187:/etc/ceph/ceph.conf 2024-01-24T13:31:17.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:16 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:17.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:16 smithi029 ceph-mon[40651]: Updating smithi163:/var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-24T13:31:17.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:16 smithi029 ceph-mon[40651]: Updating smithi187:/var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-24T13:31:17.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:16 smithi029 ceph-mon[40651]: Updating smithi029:/var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-24T13:31:17.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:16 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:17.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:16 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:17.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:16 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:17.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:16 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:17.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:16 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:17.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:16 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:17.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:16 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:17.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:16 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:17.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:16 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:17.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:16 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:17.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:16 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:17.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:16 smithi029 ceph-mon[40651]: from='client.? 172.21.15.187:0/2081034883' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-24T13:31:17.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:16 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-24T13:31:17.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:16 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-24T13:31:17.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:16 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:17.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:16 smithi163 ceph-mon[44969]: Updating smithi029:/etc/ceph/ceph.conf 2024-01-24T13:31:17.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:16 smithi163 ceph-mon[44969]: Updating smithi163:/etc/ceph/ceph.conf 2024-01-24T13:31:17.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:16 smithi163 ceph-mon[44969]: Updating smithi187:/etc/ceph/ceph.conf 2024-01-24T13:31:17.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:16 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:17.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:16 smithi163 ceph-mon[44969]: Updating smithi163:/var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-24T13:31:17.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:16 smithi163 ceph-mon[44969]: Updating smithi187:/var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-24T13:31:17.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:16 smithi163 ceph-mon[44969]: Updating smithi029:/var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-24T13:31:17.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:16 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:17.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:16 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:17.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:16 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:17.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:16 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:17.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:16 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:17.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:16 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:17.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:16 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:17.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:16 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:17.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:16 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:17.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:16 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:17.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:16 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:17.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:16 smithi163 ceph-mon[44969]: from='client.? 172.21.15.187:0/2081034883' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-24T13:31:17.234 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:16 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-24T13:31:17.234 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:16 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-24T13:31:17.234 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:16 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:17.367 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T13:31:18.060 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:17 smithi163 ceph-mon[44969]: Reconfiguring mon.a (unknown last config time)... 2024-01-24T13:31:18.060 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:17 smithi163 ceph-mon[44969]: Reconfiguring daemon mon.a on smithi029 2024-01-24T13:31:18.060 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:17 smithi163 ceph-mon[44969]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:18.060 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:17 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:18.060 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:17 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:18.060 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:17 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-24T13:31:18.060 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:17 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-24T13:31:18.060 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:17 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:18.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:17 smithi029 ceph-mon[40651]: Reconfiguring mon.a (unknown last config time)... 2024-01-24T13:31:18.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:17 smithi029 ceph-mon[40651]: Reconfiguring daemon mon.a on smithi029 2024-01-24T13:31:18.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:17 smithi029 ceph-mon[40651]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:18.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:17 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:18.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:17 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:18.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:17 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-24T13:31:18.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:17 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-24T13:31:18.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:17 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:18.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:17 smithi187 ceph-mon[44911]: Reconfiguring mon.a (unknown last config time)... 2024-01-24T13:31:18.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:17 smithi187 ceph-mon[44911]: Reconfiguring daemon mon.a on smithi029 2024-01-24T13:31:18.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:17 smithi187 ceph-mon[44911]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:18.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:17 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:18.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:17 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:18.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:17 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-24T13:31:18.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:17 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-24T13:31:18.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:17 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:18.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.346+0000 7fb188e86700 1 -- 172.21.15.29:0/4241552453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb184069450 msgr2=0x7fb184069830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:31:18.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.346+0000 7fb188e86700 1 --2- 172.21.15.29:0/4241552453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb184069450 0x7fb184069830 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb1740099c0 tx=0x7fb174009cd0 comp rx=0 tx=0).stop 2024-01-24T13:31:18.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.347+0000 7fb188e86700 1 -- 172.21.15.29:0/4241552453 shutdown_connections 2024-01-24T13:31:18.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.347+0000 7fb188e86700 1 --2- 172.21.15.29:0/4241552453 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb184103110 0x7fb18410dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:18.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.347+0000 7fb188e86700 1 --2- 172.21.15.29:0/4241552453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb184069450 0x7fb184069830 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:18.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.347+0000 7fb188e86700 1 --2- 172.21.15.29:0/4241552453 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb184069d70 0x7fb1841029e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:18.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.347+0000 7fb188e86700 1 -- 172.21.15.29:0/4241552453 >> 172.21.15.29:0/4241552453 conn(0x7fb184076d30 msgr2=0x7fb184077140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:31:18.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.347+0000 7fb188e86700 1 -- 172.21.15.29:0/4241552453 shutdown_connections 2024-01-24T13:31:18.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.347+0000 7fb188e86700 1 -- 172.21.15.29:0/4241552453 wait complete. 2024-01-24T13:31:18.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.347+0000 7fb188e86700 1 Processor -- start 2024-01-24T13:31:18.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.348+0000 7fb188e86700 1 -- start start 2024-01-24T13:31:18.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.348+0000 7fb188e86700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb184069450 0x7fb184199320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:31:18.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.348+0000 7fb188e86700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb184069d70 0x7fb184199860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:31:18.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.348+0000 7fb188e86700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb184103110 0x7fb18419dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:31:18.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.348+0000 7fb188e86700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb18419e2e0 con 0x7fb184069450 2024-01-24T13:31:18.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.348+0000 7fb188e86700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb18419e450 con 0x7fb184069d70 2024-01-24T13:31:18.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.348+0000 7fb188e86700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb18419e610 con 0x7fb184103110 2024-01-24T13:31:18.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.348+0000 7fb18259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb184069450 0x7fb184199320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:31:18.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.348+0000 7fb18259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb184069450 0x7fb184199320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59312/0 (socket says 172.21.15.29:59312) 2024-01-24T13:31:18.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.348+0000 7fb18259c700 1 -- 172.21.15.29:0/2135133624 learned_addr learned my addr 172.21.15.29:0/2135133624 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:31:18.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.348+0000 7fb182d9d700 1 --2- 172.21.15.29:0/2135133624 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb184103110 0x7fb18419dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:31:18.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.348+0000 7fb179d9b700 1 --2- 172.21.15.29:0/2135133624 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb184069d70 0x7fb184199860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:31:18.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.348+0000 7fb18259c700 1 -- 172.21.15.29:0/2135133624 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb184103110 msgr2=0x7fb18419dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:31:18.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.348+0000 7fb18259c700 1 --2- 172.21.15.29:0/2135133624 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb184103110 0x7fb18419dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:18.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.348+0000 7fb18259c700 1 -- 172.21.15.29:0/2135133624 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb184069d70 msgr2=0x7fb184199860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:31:18.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.348+0000 7fb18259c700 1 --2- 172.21.15.29:0/2135133624 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb184069d70 0x7fb184199860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:18.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.348+0000 7fb18259c700 1 -- 172.21.15.29:0/2135133624 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1740095f0 con 0x7fb184069450 2024-01-24T13:31:18.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.348+0000 7fb179d9b700 1 --2- 172.21.15.29:0/2135133624 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb184069d70 0x7fb184199860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T13:31:18.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.349+0000 7fb18259c700 1 --2- 172.21.15.29:0/2135133624 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb184069450 0x7fb184199320 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb174009990 tx=0x7fb17400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:31:18.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.349+0000 7fb17bfff700 1 -- 172.21.15.29:0/2135133624 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17401d070 con 0x7fb184069450 2024-01-24T13:31:18.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.349+0000 7fb188e86700 1 -- 172.21.15.29:0/2135133624 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb18419e890 con 0x7fb184069450 2024-01-24T13:31:18.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.349+0000 7fb17bfff700 1 -- 172.21.15.29:0/2135133624 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb1740054f0 con 0x7fb184069450 2024-01-24T13:31:18.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.349+0000 7fb17bfff700 1 -- 172.21.15.29:0/2135133624 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb174004ae0 con 0x7fb184069450 2024-01-24T13:31:18.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.349+0000 7fb188e86700 1 -- 172.21.15.29:0/2135133624 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1841a6150 con 0x7fb184069450 2024-01-24T13:31:18.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.350+0000 7fb17bfff700 1 -- 172.21.15.29:0/2135133624 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45324+0+0 (secure 0 0 0) 0x7fb174004ca0 con 0x7fb184069450 2024-01-24T13:31:18.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.350+0000 7fb188e86700 1 -- 172.21.15.29:0/2135133624 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb18404ea40 con 0x7fb184069450 2024-01-24T13:31:18.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.350+0000 7fb17bfff700 1 --2- 172.21.15.29:0/2135133624 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb1700385d0 0x7fb17003aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:31:18.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.350+0000 7fb17bfff700 1 -- 172.21.15.29:0/2135133624 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb17404bb70 con 0x7fb184069450 2024-01-24T13:31:18.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.350+0000 7fb179d9b700 1 --2- 172.21.15.29:0/2135133624 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb1700385d0 0x7fb17003aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:31:18.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.351+0000 7fb179d9b700 1 --2- 172.21.15.29:0/2135133624 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb1700385d0 0x7fb17003aa90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb16c005950 tx=0x7fb16c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:31:18.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.355+0000 7fb17bfff700 1 -- 172.21.15.29:0/2135133624 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fb174029770 con 0x7fb184069450 2024-01-24T13:31:18.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.560+0000 7fb188e86700 1 -- 172.21.15.29:0/2135133624 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fb18419a740 con 0x7fb184069450 2024-01-24T13:31:18.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.561+0000 7fb17bfff700 1 -- 172.21.15.29:0/2135133624 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+275 (secure 0 0 0) 0x7fb174026030 con 0x7fb184069450 2024-01-24T13:31:18.561 INFO:teuthology.orchestra.run.smithi029.stdout:# minimal ceph.conf for 83afcf5e-babc-11ee-95b1-87774f69a715 2024-01-24T13:31:18.561 INFO:teuthology.orchestra.run.smithi029.stdout:[global] 2024-01-24T13:31:18.561 INFO:teuthology.orchestra.run.smithi029.stdout: fsid = 83afcf5e-babc-11ee-95b1-87774f69a715 2024-01-24T13:31:18.561 INFO:teuthology.orchestra.run.smithi029.stdout: mon_host = [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] 2024-01-24T13:31:18.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.563+0000 7fb188e86700 1 -- 172.21.15.29:0/2135133624 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb1700385d0 msgr2=0x7fb17003aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:31:18.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.563+0000 7fb188e86700 1 --2- 172.21.15.29:0/2135133624 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb1700385d0 0x7fb17003aa90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb16c005950 tx=0x7fb16c0058e0 comp rx=0 tx=0).stop 2024-01-24T13:31:18.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.563+0000 7fb188e86700 1 -- 172.21.15.29:0/2135133624 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb184069450 msgr2=0x7fb184199320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:31:18.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.563+0000 7fb188e86700 1 --2- 172.21.15.29:0/2135133624 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb184069450 0x7fb184199320 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb174009990 tx=0x7fb17400f690 comp rx=0 tx=0).stop 2024-01-24T13:31:18.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.563+0000 7fb188e86700 1 -- 172.21.15.29:0/2135133624 shutdown_connections 2024-01-24T13:31:18.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.563+0000 7fb188e86700 1 --2- 172.21.15.29:0/2135133624 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb184103110 0x7fb18419dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:18.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.563+0000 7fb188e86700 1 --2- 172.21.15.29:0/2135133624 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb1700385d0 0x7fb17003aa90 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:18.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.563+0000 7fb188e86700 1 --2- 172.21.15.29:0/2135133624 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb184069450 0x7fb184199320 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:18.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.563+0000 7fb188e86700 1 --2- 172.21.15.29:0/2135133624 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb184069d70 0x7fb184199860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:18.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.563+0000 7fb188e86700 1 -- 172.21.15.29:0/2135133624 >> 172.21.15.29:0/2135133624 conn(0x7fb184076d30 msgr2=0x7fb18410c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:31:18.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.563+0000 7fb188e86700 1 -- 172.21.15.29:0/2135133624 shutdown_connections 2024-01-24T13:31:18.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:18.563+0000 7fb188e86700 1 -- 172.21.15.29:0/2135133624 wait complete. 2024-01-24T13:31:18.979 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-01-24T13:31:18.979 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-24T13:31:18.979 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.conf 2024-01-24T13:31:19.017 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-24T13:31:19.017 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-24T13:31:19.093 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-01-24T13:31:19.093 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/ceph/ceph.conf 2024-01-24T13:31:19.131 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-01-24T13:31:19.131 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-24T13:31:19.200 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:18 smithi163 ceph-mon[44969]: Reconfiguring mon.b (monmap changed)... 2024-01-24T13:31:19.200 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:18 smithi163 ceph-mon[44969]: Reconfiguring daemon mon.b on smithi163 2024-01-24T13:31:19.200 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:18 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:19.200 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:18 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:19.200 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:18 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-24T13:31:19.200 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:18 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-24T13:31:19.200 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:18 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:19.200 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:18 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/2135133624' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:19.202 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-01-24T13:31:19.202 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/ceph/ceph.conf 2024-01-24T13:31:19.227 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:18 smithi187 ceph-mon[44911]: Reconfiguring mon.b (monmap changed)... 2024-01-24T13:31:19.227 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:18 smithi187 ceph-mon[44911]: Reconfiguring daemon mon.b on smithi163 2024-01-24T13:31:19.227 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:18 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:19.227 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:18 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:19.227 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:18 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-24T13:31:19.227 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:18 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-24T13:31:19.227 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:18 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:19.227 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:18 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/2135133624' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:19.232 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-01-24T13:31:19.232 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-24T13:31:19.301 INFO:tasks.cephadm:Adding mgr.x on smithi029 2024-01-24T13:31:19.301 INFO:tasks.cephadm:Adding mgr.y on smithi163 2024-01-24T13:31:19.301 INFO:tasks.cephadm:Adding mgr.z on smithi187 2024-01-24T13:31:19.302 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph orch apply mgr '3;smithi029=x;smithi163=y;smithi187=z' 2024-01-24T13:31:19.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:18 smithi029 ceph-mon[40651]: Reconfiguring mon.b (monmap changed)... 2024-01-24T13:31:19.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:18 smithi029 ceph-mon[40651]: Reconfiguring daemon mon.b on smithi163 2024-01-24T13:31:19.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:18 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:19.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:18 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:19.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:18 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-24T13:31:19.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:18 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-24T13:31:19.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:18 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:19.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:18 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/2135133624' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:19.555 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.c/config 2024-01-24T13:31:20.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:19 smithi163 ceph-mon[44969]: Reconfiguring mon.c (monmap changed)... 2024-01-24T13:31:20.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:19 smithi163 ceph-mon[44969]: Reconfiguring daemon mon.c on smithi187 2024-01-24T13:31:20.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:19 smithi163 ceph-mon[44969]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:20.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:19 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:20.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:19 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:20.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:19 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:31:20.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:19 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:20.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:19 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:31:20.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:19 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:20.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:19 smithi187 ceph-mon[44911]: Reconfiguring mon.c (monmap changed)... 2024-01-24T13:31:20.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:19 smithi187 ceph-mon[44911]: Reconfiguring daemon mon.c on smithi187 2024-01-24T13:31:20.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:19 smithi187 ceph-mon[44911]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:20.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:19 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:20.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:19 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:20.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:19 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:31:20.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:19 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:20.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:19 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:31:20.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:19 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:20.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:19 smithi029 ceph-mon[40651]: Reconfiguring mon.c (monmap changed)... 2024-01-24T13:31:20.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:19 smithi029 ceph-mon[40651]: Reconfiguring daemon mon.c on smithi187 2024-01-24T13:31:20.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:19 smithi029 ceph-mon[40651]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:20.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:19 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:20.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:19 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:20.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:19 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:31:20.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:19 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:20.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:19 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:31:20.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:19 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:20.529 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.526+0000 7f2e84d19700 1 -- 172.21.15.187:0/20512101 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e80106ac0 msgr2=0x7f2e80108f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:31:20.529 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.526+0000 7f2e84d19700 1 --2- 172.21.15.187:0/20512101 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e80106ac0 0x7f2e80108f30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2e7800b0a0 tx=0x7f2e7800b3b0 comp rx=0 tx=0).stop 2024-01-24T13:31:20.529 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.526+0000 7f2e84d19700 1 -- 172.21.15.187:0/20512101 shutdown_connections 2024-01-24T13:31:20.529 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.526+0000 7f2e84d19700 1 --2- 172.21.15.187:0/20512101 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e80106ac0 0x7f2e80108f30 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:20.529 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.526+0000 7f2e84d19700 1 --2- 172.21.15.187:0/20512101 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e80101860 0x7f2e80103c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:20.529 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.526+0000 7f2e84d19700 1 --2- 172.21.15.187:0/20512101 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2e80104190 0x7f2e80106580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:20.529 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.526+0000 7f2e84d19700 1 -- 172.21.15.187:0/20512101 >> 172.21.15.187:0/20512101 conn(0x7f2e800fb250 msgr2=0x7f2e800fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:31:20.530 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.527+0000 7f2e84d19700 1 -- 172.21.15.187:0/20512101 shutdown_connections 2024-01-24T13:31:20.530 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.527+0000 7f2e84d19700 1 -- 172.21.15.187:0/20512101 wait complete. 2024-01-24T13:31:20.530 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.527+0000 7f2e84d19700 1 Processor -- start 2024-01-24T13:31:20.530 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.527+0000 7f2e84d19700 1 -- start start 2024-01-24T13:31:20.530 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.527+0000 7f2e84d19700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e80101860 0x7f2e80196ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:31:20.530 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.527+0000 7f2e84d19700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e80104190 0x7f2e80197010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:31:20.530 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.527+0000 7f2e84d19700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2e80106ac0 0x7f2e8019e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:31:20.530 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.527+0000 7f2e84d19700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e8019e660 con 0x7f2e80104190 2024-01-24T13:31:20.530 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.527+0000 7f2e84d19700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e8019e7a0 con 0x7f2e80106ac0 2024-01-24T13:31:20.530 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.527+0000 7f2e84d19700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e8019e930 con 0x7f2e80101860 2024-01-24T13:31:20.530 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.528+0000 7f2e7e59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e80101860 0x7f2e80196ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:31:20.530 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.528+0000 7f2e7e59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e80101860 0x7f2e80196ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:48256/0 (socket says 172.21.15.187:48256) 2024-01-24T13:31:20.530 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.528+0000 7f2e7e59c700 1 -- 172.21.15.187:0/1368011682 learned_addr learned my addr 172.21.15.187:0/1368011682 (peer_addr_for_me v2:172.21.15.187:0/0) 2024-01-24T13:31:20.530 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.528+0000 7f2e7ed9d700 1 --2- 172.21.15.187:0/1368011682 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2e80106ac0 0x7f2e8019e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:31:20.531 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.528+0000 7f2e7dd9b700 1 --2- 172.21.15.187:0/1368011682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e80104190 0x7f2e80197010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:31:20.531 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.528+0000 7f2e7e59c700 1 -- 172.21.15.187:0/1368011682 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2e80106ac0 msgr2=0x7f2e8019e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:31:20.531 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.528+0000 7f2e7e59c700 1 --2- 172.21.15.187:0/1368011682 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2e80106ac0 0x7f2e8019e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:20.531 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.528+0000 7f2e7e59c700 1 -- 172.21.15.187:0/1368011682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e80104190 msgr2=0x7f2e80197010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:31:20.531 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.528+0000 7f2e7e59c700 1 --2- 172.21.15.187:0/1368011682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e80104190 0x7f2e80197010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:20.531 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.528+0000 7f2e7e59c700 1 -- 172.21.15.187:0/1368011682 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e78009d00 con 0x7f2e80101860 2024-01-24T13:31:20.531 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.528+0000 7f2e7dd9b700 1 --2- 172.21.15.187:0/1368011682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e80104190 0x7f2e80197010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T13:31:20.531 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.528+0000 7f2e7e59c700 1 --2- 172.21.15.187:0/1368011682 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e80101860 0x7f2e80196ad0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2e6800c9e0 tx=0x7f2e6800cda0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:31:20.531 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.528+0000 7f2e777fe700 1 -- 172.21.15.187:0/1368011682 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e68007ac0 con 0x7f2e80101860 2024-01-24T13:31:20.531 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.528+0000 7f2e84d19700 1 -- 172.21.15.187:0/1368011682 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e8019ec10 con 0x7f2e80101860 2024-01-24T13:31:20.531 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.528+0000 7f2e777fe700 1 -- 172.21.15.187:0/1368011682 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2e68007c20 con 0x7f2e80101860 2024-01-24T13:31:20.531 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.528+0000 7f2e777fe700 1 -- 172.21.15.187:0/1368011682 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e68018660 con 0x7f2e80101860 2024-01-24T13:31:20.531 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.528+0000 7f2e84d19700 1 -- 172.21.15.187:0/1368011682 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e8019f110 con 0x7f2e80101860 2024-01-24T13:31:20.532 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.529+0000 7f2e777fe700 1 -- 172.21.15.187:0/1368011682 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 13) v1 ==== 45324+0+0 (secure 0 0 0) 0x7f2e680187c0 con 0x7f2e80101860 2024-01-24T13:31:20.532 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.529+0000 7f2e777fe700 1 --2- 172.21.15.187:0/1368011682 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f2e6c0385d0 0x7f2e6c03aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:31:20.532 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.529+0000 7f2e777fe700 1 -- 172.21.15.187:0/1368011682 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2e6804ad20 con 0x7f2e80101860 2024-01-24T13:31:20.532 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.529+0000 7f2e84d19700 1 -- 172.21.15.187:0/1368011682 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e800fcdd0 con 0x7f2e80101860 2024-01-24T13:31:20.533 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.530+0000 7f2e7dd9b700 1 --2- 172.21.15.187:0/1368011682 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f2e6c0385d0 0x7f2e6c03aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:31:20.533 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.530+0000 7f2e7dd9b700 1 --2- 172.21.15.187:0/1368011682 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f2e6c0385d0 0x7f2e6c03aa90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2e70005fd0 tx=0x7f2e70005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:31:20.535 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.534+0000 7f2e777fe700 1 -- 172.21.15.187:0/1368011682 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f2e68018a70 con 0x7f2e80101860 2024-01-24T13:31:20.733 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.732+0000 7f2e84d19700 1 -- 172.21.15.187:0/1368011682 --> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi029=x;smithi163=y;smithi187=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e80061180 con 0x7f2e6c0385d0 2024-01-24T13:31:20.739 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.739+0000 7f2e777fe700 1 -- 172.21.15.187:0/1368011682 <== mgr.14152 v2:172.21.15.29:6800/2498991338 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f2e80061180 con 0x7f2e6c0385d0 2024-01-24T13:31:20.739 INFO:teuthology.orchestra.run.smithi187.stdout:Scheduled mgr update... 2024-01-24T13:31:20.741 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.741+0000 7f2e84d19700 1 -- 172.21.15.187:0/1368011682 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f2e6c0385d0 msgr2=0x7f2e6c03aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:31:20.742 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.741+0000 7f2e84d19700 1 --2- 172.21.15.187:0/1368011682 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f2e6c0385d0 0x7f2e6c03aa90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2e70005fd0 tx=0x7f2e70005dc0 comp rx=0 tx=0).stop 2024-01-24T13:31:20.742 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.741+0000 7f2e84d19700 1 -- 172.21.15.187:0/1368011682 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e80101860 msgr2=0x7f2e80196ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:31:20.742 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.741+0000 7f2e84d19700 1 --2- 172.21.15.187:0/1368011682 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e80101860 0x7f2e80196ad0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2e6800c9e0 tx=0x7f2e6800cda0 comp rx=0 tx=0).stop 2024-01-24T13:31:20.742 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.741+0000 7f2e84d19700 1 -- 172.21.15.187:0/1368011682 shutdown_connections 2024-01-24T13:31:20.742 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.741+0000 7f2e84d19700 1 --2- 172.21.15.187:0/1368011682 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2e80106ac0 0x7f2e8019e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:20.742 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.741+0000 7f2e84d19700 1 --2- 172.21.15.187:0/1368011682 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e80101860 0x7f2e80196ad0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:20.742 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.741+0000 7f2e84d19700 1 --2- 172.21.15.187:0/1368011682 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f2e6c0385d0 0x7f2e6c03aa90 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:20.742 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.741+0000 7f2e84d19700 1 --2- 172.21.15.187:0/1368011682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e80104190 0x7f2e80197010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:20.742 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.741+0000 7f2e84d19700 1 -- 172.21.15.187:0/1368011682 >> 172.21.15.187:0/1368011682 conn(0x7f2e800fb250 msgr2=0x7f2e801024f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:31:20.742 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.741+0000 7f2e84d19700 1 -- 172.21.15.187:0/1368011682 shutdown_connections 2024-01-24T13:31:20.742 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:31:20.741+0000 7f2e84d19700 1 -- 172.21.15.187:0/1368011682 wait complete. 2024-01-24T13:31:21.121 DEBUG:teuthology.orchestra.run.smithi163:mgr.y> sudo journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mgr.y.service 2024-01-24T13:31:21.124 DEBUG:teuthology.orchestra.run.smithi187:mgr.z> sudo journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mgr.z.service 2024-01-24T13:31:21.126 INFO:tasks.cephadm:Deploying OSDs... 2024-01-24T13:31:21.126 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-24T13:31:21.126 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/scratch_devs of=/dev/stdout 2024-01-24T13:31:21.148 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-24T13:31:21.148 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_1 2024-01-24T13:31:21.156 INFO:journalctl@ceph.mgr.z.smithi187.stdout:-- Logs begin at Wed 2024-01-24 13:13:23 UTC. -- 2024-01-24T13:31:21.157 INFO:journalctl@ceph.mgr.y.smithi163.stdout:-- Logs begin at Wed 2024-01-24 13:14:09 UTC. -- 2024-01-24T13:31:21.211 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-24T13:31:21.211 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-24T13:31:21.211 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 6h/6d Inode: 176630 Links: 1 2024-01-24T13:31:21.211 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-24T13:31:21.211 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-01-24T13:31:21.211 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-01-24 13:30:40.508954407 +0000 2024-01-24T13:31:21.212 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-01-24 13:24:29.412205954 +0000 2024-01-24T13:31:21.212 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-01-24 13:24:29.412205954 +0000 2024-01-24T13:31:21.212 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: - 2024-01-24T13:31:21.212 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-24T13:31:21.285 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-01-24T13:31:21.285 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-01-24T13:31:21.285 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000128525 s, 4.0 MB/s 2024-01-24T13:31:21.288 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-24T13:31:21.351 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_2 2024-01-24T13:31:21.412 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-24T13:31:21.413 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-24T13:31:21.413 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 6h/6d Inode: 174031 Links: 1 2024-01-24T13:31:21.413 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-24T13:31:21.413 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-01-24T13:31:21.413 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-01-24 13:30:40.562953480 +0000 2024-01-24T13:31:21.413 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-01-24 13:24:29.788199471 +0000 2024-01-24T13:31:21.413 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-01-24 13:24:29.788199471 +0000 2024-01-24T13:31:21.413 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: - 2024-01-24T13:31:21.413 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-24T13:31:21.485 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-01-24T13:31:21.485 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-01-24T13:31:21.485 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.00012265 s, 4.2 MB/s 2024-01-24T13:31:21.487 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-24T13:31:21.549 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_3 2024-01-24T13:31:21.612 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-24T13:31:21.612 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-24T13:31:21.612 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 6h/6d Inode: 176742 Links: 1 2024-01-24T13:31:21.612 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-24T13:31:21.612 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-01-24T13:31:21.612 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-01-24 13:30:40.616952553 +0000 2024-01-24T13:31:21.612 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-01-24 13:24:30.162193022 +0000 2024-01-24T13:31:21.613 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-01-24 13:24:30.162193022 +0000 2024-01-24T13:31:21.613 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: - 2024-01-24T13:31:21.613 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-24T13:31:21.686 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-01-24T13:31:21.686 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-01-24T13:31:21.686 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000124919 s, 4.1 MB/s 2024-01-24T13:31:21.688 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-24T13:31:21.752 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_4 2024-01-24T13:31:21.814 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-24T13:31:21.815 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-24T13:31:21.815 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 6h/6d Inode: 169819 Links: 1 2024-01-24T13:31:21.815 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-24T13:31:21.815 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-01-24T13:31:21.815 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-01-24 13:30:40.673951575 +0000 2024-01-24T13:31:21.815 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-01-24 13:24:30.516186919 +0000 2024-01-24T13:31:21.815 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-01-24 13:24:30.516186919 +0000 2024-01-24T13:31:21.815 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: - 2024-01-24T13:31:21.815 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-24T13:31:21.888 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-01-24T13:31:21.888 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-01-24T13:31:21.888 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000130505 s, 3.9 MB/s 2024-01-24T13:31:21.890 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-24T13:31:21.953 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-01-24T13:31:21.953 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/scratch_devs of=/dev/stdout 2024-01-24T13:31:21.974 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-24T13:31:21.975 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_1 2024-01-24T13:31:22.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:21 smithi187 ceph-mon[44911]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi029=x;smithi163=y;smithi187=z", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:31:22.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:21 smithi187 ceph-mon[44911]: Saving service mgr spec with placement smithi029=x;smithi163=y;smithi187=z;count:3 2024-01-24T13:31:22.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:21 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:22.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:21 smithi187 ceph-mon[44911]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:22.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:21 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:31:22.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:21 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:22.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:21 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:31:22.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:21 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:22.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:21 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-24T13:31:22.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:21 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-01-24T13:31:22.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:21 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-24T13:31:22.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:21 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:22.013 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:21 smithi187 ceph-mon[44911]: Deploying daemon mgr.z on smithi187 2024-01-24T13:31:22.036 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-24T13:31:22.036 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-24T13:31:22.036 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 6h/6d Inode: 171625 Links: 1 2024-01-24T13:31:22.036 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-24T13:31:22.036 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-01-24T13:31:22.036 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-01-24 13:31:09.159068893 +0000 2024-01-24T13:31:22.036 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-01-24 13:23:56.493584148 +0000 2024-01-24T13:31:22.036 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-01-24 13:23:56.493584148 +0000 2024-01-24T13:31:22.036 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: - 2024-01-24T13:31:22.037 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-24T13:31:22.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:21 smithi029 ceph-mon[40651]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi029=x;smithi163=y;smithi187=z", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:31:22.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:21 smithi029 ceph-mon[40651]: Saving service mgr spec with placement smithi029=x;smithi163=y;smithi187=z;count:3 2024-01-24T13:31:22.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:21 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:22.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:21 smithi029 ceph-mon[40651]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:22.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:21 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:31:22.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:21 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:22.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:21 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:31:22.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:21 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:22.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:21 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-24T13:31:22.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:21 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-01-24T13:31:22.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:21 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-24T13:31:22.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:21 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:22.101 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:21 smithi029 ceph-mon[40651]: Deploying daemon mgr.z on smithi187 2024-01-24T13:31:22.105 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:21 smithi163 ceph-mon[44969]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi029=x;smithi163=y;smithi187=z", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:31:22.106 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:21 smithi163 ceph-mon[44969]: Saving service mgr spec with placement smithi029=x;smithi163=y;smithi187=z;count:3 2024-01-24T13:31:22.106 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:21 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:22.106 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:21 smithi163 ceph-mon[44969]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:22.106 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:21 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:31:22.106 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:21 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:22.106 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:21 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:31:22.107 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:21 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:22.107 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:21 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-24T13:31:22.107 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:21 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-01-24T13:31:22.107 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:21 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-24T13:31:22.107 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:21 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:22.107 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:21 smithi163 ceph-mon[44969]: Deploying daemon mgr.z on smithi187 2024-01-24T13:31:22.108 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-01-24T13:31:22.108 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-01-24T13:31:22.109 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000229045 s, 2.2 MB/s 2024-01-24T13:31:22.110 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-24T13:31:22.171 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_2 2024-01-24T13:31:22.232 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-24T13:31:22.232 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-24T13:31:22.232 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 6h/6d Inode: 173788 Links: 1 2024-01-24T13:31:22.232 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-24T13:31:22.232 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-01-24T13:31:22.232 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-01-24 13:31:09.159068893 +0000 2024-01-24T13:31:22.232 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-01-24 13:23:56.817576598 +0000 2024-01-24T13:31:22.232 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-01-24 13:23:56.817576598 +0000 2024-01-24T13:31:22.232 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: - 2024-01-24T13:31:22.233 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-24T13:31:22.303 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-01-24T13:31:22.304 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-01-24T13:31:22.304 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000227096 s, 2.3 MB/s 2024-01-24T13:31:22.305 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-24T13:31:22.305 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jan 24 13:31:22 smithi187 systemd[1]: Starting Ceph mgr.z for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:31:22.366 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_3 2024-01-24T13:31:22.427 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-24T13:31:22.427 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-24T13:31:22.427 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 6h/6d Inode: 173914 Links: 1 2024-01-24T13:31:22.427 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-24T13:31:22.427 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-01-24T13:31:22.427 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-01-24 13:31:09.159068893 +0000 2024-01-24T13:31:22.427 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-01-24 13:23:57.184568047 +0000 2024-01-24T13:31:22.427 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-01-24 13:23:57.184568047 +0000 2024-01-24T13:31:22.427 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: - 2024-01-24T13:31:22.428 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-24T13:31:22.495 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-01-24T13:31:22.495 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-01-24T13:31:22.495 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000194596 s, 2.6 MB/s 2024-01-24T13:31:22.496 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-24T13:31:22.557 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_4 2024-01-24T13:31:22.615 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-24T13:31:22.615 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-24T13:31:22.615 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 6h/6d Inode: 177350 Links: 1 2024-01-24T13:31:22.615 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-24T13:31:22.616 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-01-24T13:31:22.616 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-01-24 13:31:09.159068893 +0000 2024-01-24T13:31:22.616 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-01-24 13:23:57.530559985 +0000 2024-01-24T13:31:22.616 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-01-24 13:23:57.530559985 +0000 2024-01-24T13:31:22.616 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: - 2024-01-24T13:31:22.616 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-24T13:31:22.685 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-01-24T13:31:22.685 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-01-24T13:31:22.685 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000210103 s, 2.4 MB/s 2024-01-24T13:31:22.685 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-24T13:31:22.742 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-01-24T13:31:22.742 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/scratch_devs of=/dev/stdout 2024-01-24T13:31:22.760 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-24T13:31:22.760 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_1 2024-01-24T13:31:22.804 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jan 24 13:31:22 smithi187 ceph-mgr[46810]: -- 172.21.15.187:0/368813482 <== mon.1 v2:172.21.15.187:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560264d3d1e0 con 0x560264d12c00 2024-01-24T13:31:22.804 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jan 24 13:31:22 smithi187 bash[46772]: 429761af3acf894448fd0c2e5c478d21a7a0e930bbde4a4e75a27c694bfeb926 2024-01-24T13:31:22.804 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jan 24 13:31:22 smithi187 systemd[1]: Started Ceph mgr.z for 83afcf5e-babc-11ee-95b1-87774f69a715. 2024-01-24T13:31:22.819 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-24T13:31:22.819 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-24T13:31:22.820 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 6h/6d Inode: 174598 Links: 1 2024-01-24T13:31:22.820 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-24T13:31:22.821 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2024-01-24T13:31:22.821 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2024-01-24 13:31:22.059751940 +0000 2024-01-24T13:31:22.822 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2024-01-24 13:23:50.852363368 +0000 2024-01-24T13:31:22.822 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2024-01-24 13:23:50.852363368 +0000 2024-01-24T13:31:22.822 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: - 2024-01-24T13:31:22.823 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-24T13:31:22.887 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jan 24 13:31:22 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-z[46787]: 2024-01-24T13:31:22.878+0000 7f42ed0da200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-24T13:31:22.888 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2024-01-24T13:31:22.888 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2024-01-24T13:31:22.888 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000257523 s, 2.0 MB/s 2024-01-24T13:31:22.890 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-24T13:31:22.948 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_2 2024-01-24T13:31:23.007 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-24T13:31:23.007 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-24T13:31:23.007 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 6h/6d Inode: 174673 Links: 1 2024-01-24T13:31:23.007 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-24T13:31:23.007 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2024-01-24T13:31:23.007 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2024-01-24 13:31:22.059751940 +0000 2024-01-24T13:31:23.007 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2024-01-24 13:23:51.219357908 +0000 2024-01-24T13:31:23.007 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2024-01-24 13:23:51.219357908 +0000 2024-01-24T13:31:23.008 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: - 2024-01-24T13:31:23.008 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-24T13:31:23.074 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2024-01-24T13:31:23.075 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2024-01-24T13:31:23.075 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000241512 s, 2.1 MB/s 2024-01-24T13:31:23.076 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-24T13:31:23.134 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_3 2024-01-24T13:31:23.159 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jan 24 13:31:22 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-z[46787]: 2024-01-24T13:31:22.975+0000 7f42ed0da200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-24T13:31:23.159 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jan 24 13:31:23 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-z[46787]: 2024-01-24T13:31:23.069+0000 7f42ed0da200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-24T13:31:23.159 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jan 24 13:31:23 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-z[46787]: 2024-01-24T13:31:23.158+0000 7f42ed0da200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-24T13:31:23.176 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-24T13:31:23.176 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-24T13:31:23.176 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 6h/6d Inode: 175424 Links: 1 2024-01-24T13:31:23.177 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-24T13:31:23.177 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2024-01-24T13:31:23.177 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2024-01-24 13:31:22.059751940 +0000 2024-01-24T13:31:23.177 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2024-01-24 13:23:51.565352760 +0000 2024-01-24T13:31:23.177 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2024-01-24 13:23:51.565352760 +0000 2024-01-24T13:31:23.177 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: - 2024-01-24T13:31:23.177 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-24T13:31:23.243 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2024-01-24T13:31:23.243 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2024-01-24T13:31:23.243 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000195689 s, 2.6 MB/s 2024-01-24T13:31:23.244 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-24T13:31:23.302 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_4 2024-01-24T13:31:23.360 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-24T13:31:23.360 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-24T13:31:23.360 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 6h/6d Inode: 174857 Links: 1 2024-01-24T13:31:23.360 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-24T13:31:23.360 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2024-01-24T13:31:23.360 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2024-01-24 13:31:22.059751940 +0000 2024-01-24T13:31:23.360 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2024-01-24 13:23:51.921347463 +0000 2024-01-24T13:31:23.360 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2024-01-24 13:23:51.921347463 +0000 2024-01-24T13:31:23.360 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: - 2024-01-24T13:31:23.361 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-24T13:31:23.426 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2024-01-24T13:31:23.427 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2024-01-24T13:31:23.427 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000230925 s, 2.2 MB/s 2024-01-24T13:31:23.428 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-24T13:31:23.485 INFO:tasks.cephadm:Deploying osd.0 on smithi029 with /dev/vg_nvme/lv_4... 2024-01-24T13:31:23.486 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-01-24T13:31:23.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:23 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:23.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:23 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:23.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:23 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:23.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:23 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-24T13:31:23.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:23 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-01-24T13:31:23.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:23 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-24T13:31:23.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:23 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:23.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:23 smithi163 ceph-mon[44969]: Deploying daemon mgr.y on smithi163 2024-01-24T13:31:23.718 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:23 smithi163 ceph-mon[44969]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:23.736 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T13:31:23.804 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:23 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:23.804 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:23 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:23.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:23 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:23.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:23 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-24T13:31:23.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:23 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-01-24T13:31:23.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:23 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-24T13:31:23.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:23 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:23.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:23 smithi187 ceph-mon[44911]: Deploying daemon mgr.y on smithi163 2024-01-24T13:31:23.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:23 smithi187 ceph-mon[44911]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:23.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:23 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:23.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:23 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:23.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:23 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:23.841 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:23 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-24T13:31:23.841 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:23 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-01-24T13:31:23.841 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:23 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-24T13:31:23.841 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:23 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:23.841 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:23 smithi029 ceph-mon[40651]: Deploying daemon mgr.y on smithi163 2024-01-24T13:31:23.841 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:23 smithi029 ceph-mon[40651]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:23.982 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Jan 24 13:31:23 smithi163 systemd[1]: Starting Ceph mgr.y for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:31:24.122 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jan 24 13:31:23 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-z[46787]: 2024-01-24T13:31:23.805+0000 7f42ed0da200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-24T13:31:24.122 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jan 24 13:31:24 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-z[46787]: 2024-01-24T13:31:24.038+0000 7f42ed0da200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-24T13:31:24.387 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jan 24 13:31:24 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-z[46787]: 2024-01-24T13:31:24.121+0000 7f42ed0da200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-24T13:31:24.387 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jan 24 13:31:24 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-z[46787]: 2024-01-24T13:31:24.228+0000 7f42ed0da200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-24T13:31:24.482 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Jan 24 13:31:24 smithi163 ceph-mgr[46479]: -- 172.21.15.163:0/972074150 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5629117431e0 con 0x562911718800 2024-01-24T13:31:24.482 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Jan 24 13:31:24 smithi163 bash[46441]: c4b0cf59f7bb22fba399c8df40f1a393a42fa1273ea9823850039a922b27010e 2024-01-24T13:31:24.482 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Jan 24 13:31:24 smithi163 systemd[1]: Started Ceph mgr.y for 83afcf5e-babc-11ee-95b1-87774f69a715. 2024-01-24T13:31:24.804 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jan 24 13:31:24 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-z[46787]: 2024-01-24T13:31:24.386+0000 7f42ed0da200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-24T13:31:24.805 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jan 24 13:31:24 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-z[46787]: 2024-01-24T13:31:24.513+0000 7f42ed0da200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-24T13:31:24.805 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jan 24 13:31:24 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-z[46787]: 2024-01-24T13:31:24.606+0000 7f42ed0da200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-24T13:31:24.894 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Jan 24 13:31:24 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-y[46456]: 2024-01-24T13:31:24.608+0000 7f424811c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-24T13:31:24.894 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Jan 24 13:31:24 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-y[46456]: 2024-01-24T13:31:24.705+0000 7f424811c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-24T13:31:24.894 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Jan 24 13:31:24 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-y[46456]: 2024-01-24T13:31:24.799+0000 7f424811c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-24T13:31:24.894 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Jan 24 13:31:24 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-y[46456]: 2024-01-24T13:31:24.892+0000 7f424811c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-24T13:31:25.066 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-01-24T13:31:25.066 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-01-24T13:31:25.066 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10+0 records in 2024-01-24T13:31:25.066 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2024-01-24T13:31:25.066 INFO:teuthology.orchestra.run.smithi029.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00942429 s, 1.1 GB/s 2024-01-24T13:31:25.066 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping successful for: 2024-01-24T13:31:25.304 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jan 24 13:31:25 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-z[46787]: 2024-01-24T13:31:25.013+0000 7f42ed0da200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-24T13:31:25.453 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph orch daemon add osd smithi029:vg_nvme/lv_4 2024-01-24T13:31:25.610 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jan 24 13:31:25 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-z[46787]: 2024-01-24T13:31:25.311+0000 7f42ed0da200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-24T13:31:25.610 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:25 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:25.610 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:25 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:25.610 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:25 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:25.610 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:25 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:25.610 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:25 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:31:25.610 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:25 smithi187 ceph-mon[44911]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:25.610 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:25 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:25.610 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:25 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:25.705 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T13:31:25.732 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Jan 24 13:31:25 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-y[46456]: 2024-01-24T13:31:25.551+0000 7f424811c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-24T13:31:25.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:25 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:25.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:25 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:25.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:25 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:25.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:25 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:25.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:25 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:31:25.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:25 smithi163 ceph-mon[44969]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:25.734 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:25 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:25.734 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:25 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:25.824 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:25 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:25.824 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:25 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:25.824 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:25 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:25.824 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:25 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:25.825 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:25 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:31:25.825 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:25 smithi029 ceph-mon[40651]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:25.825 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:25 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:25.825 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:25 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:26.054 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jan 24 13:31:25 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-z[46787]: 2024-01-24T13:31:25.609+0000 7f42ed0da200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-24T13:31:26.054 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jan 24 13:31:25 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-z[46787]: 2024-01-24T13:31:25.700+0000 7f42ed0da200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-24T13:31:26.137 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Jan 24 13:31:25 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-y[46456]: 2024-01-24T13:31:25.786+0000 7f424811c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-24T13:31:26.137 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Jan 24 13:31:25 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-y[46456]: 2024-01-24T13:31:25.871+0000 7f424811c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-24T13:31:26.137 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Jan 24 13:31:25 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-y[46456]: 2024-01-24T13:31:25.978+0000 7f424811c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-24T13:31:26.137 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Jan 24 13:31:26 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-y[46456]: 2024-01-24T13:31:26.135+0000 7f424811c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-24T13:31:26.387 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Jan 24 13:31:26 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-y[46456]: 2024-01-24T13:31:26.261+0000 7f424811c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-24T13:31:26.387 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Jan 24 13:31:26 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-y[46456]: 2024-01-24T13:31:26.345+0000 7f424811c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-24T13:31:26.387 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:26 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:26.387 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:26 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:26.387 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:26 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:26.387 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:26 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:31:26.387 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:26 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:26.387 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:26 smithi163 ceph-mon[44969]: Reconfiguring mgr.x (unknown last config time)... 2024-01-24T13:31:26.387 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:26 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-24T13:31:26.388 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:26 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-24T13:31:26.388 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:26 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:26.706 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jan 24 13:31:26 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-z[46787]: 2024-01-24T13:31:26.466+0000 7f42ed0da200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-24T13:31:26.706 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jan 24 13:31:26 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-z[46787]: 2024-01-24T13:31:26.560+0000 7f42ed0da200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-24T13:31:26.707 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:26 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:26.707 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:26 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:26.707 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:26 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:26.707 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:26 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:31:26.707 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:26 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:26.707 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:26 smithi187 ceph-mon[44911]: Reconfiguring mgr.x (unknown last config time)... 2024-01-24T13:31:26.707 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:26 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-24T13:31:26.707 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:26 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-24T13:31:26.707 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:26 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:26.707 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:26 smithi187 ceph-mon[44911]: Reconfiguring daemon mgr.x on smithi029 2024-01-24T13:31:26.707 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:26 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:26.717 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:26 smithi163 ceph-mon[44969]: Reconfiguring daemon mgr.x on smithi029 2024-01-24T13:31:26.717 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:26 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:26.719 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:26 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:26.719 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:26 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:26.720 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:26 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:26.720 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:26 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:31:26.720 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:26 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:26.720 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:26 smithi029 ceph-mon[40651]: Reconfiguring mgr.x (unknown last config time)... 2024-01-24T13:31:26.720 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:26 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-24T13:31:26.720 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:26 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-24T13:31:26.720 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:26 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:26.720 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:26 smithi029 ceph-mon[40651]: Reconfiguring daemon mgr.x on smithi029 2024-01-24T13:31:26.720 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:26 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:26.981 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Jan 24 13:31:26 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-y[46456]: 2024-01-24T13:31:26.716+0000 7f424811c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-24T13:31:26.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:26.991+0000 7f0c32f93700 1 -- 172.21.15.29:0/3418051190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c2c101870 msgr2=0x7f0c2c108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:31:26.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:26.991+0000 7f0c32f93700 1 --2- 172.21.15.29:0/3418051190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c2c101870 0x7f0c2c108120 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0c2400b0d0 tx=0x7f0c2400b3e0 comp rx=0 tx=0).stop 2024-01-24T13:31:26.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:26.991+0000 7f0c32f93700 1 -- 172.21.15.29:0/3418051190 shutdown_connections 2024-01-24T13:31:26.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:26.991+0000 7f0c32f93700 1 --2- 172.21.15.29:0/3418051190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c2c101870 0x7f0c2c108120 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:26.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:26.991+0000 7f0c32f93700 1 --2- 172.21.15.29:0/3418051190 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c2c0ffcd0 0x7f0c2c1000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:26.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:26.991+0000 7f0c32f93700 1 --2- 172.21.15.29:0/3418051190 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0c2c100ed0 0x7f0c2c101330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:26.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:26.991+0000 7f0c32f93700 1 -- 172.21.15.29:0/3418051190 >> 172.21.15.29:0/3418051190 conn(0x7f0c2c0fb250 msgr2=0x7f0c2c0fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:31:26.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:26.991+0000 7f0c32f93700 1 -- 172.21.15.29:0/3418051190 shutdown_connections 2024-01-24T13:31:26.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:26.991+0000 7f0c32f93700 1 -- 172.21.15.29:0/3418051190 wait complete. 2024-01-24T13:31:26.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:26.992+0000 7f0c32f93700 1 Processor -- start 2024-01-24T13:31:26.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:26.992+0000 7f0c32f93700 1 -- start start 2024-01-24T13:31:26.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:26.992+0000 7f0c32f93700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c2c0ffcd0 0x7f0c2c198da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:31:26.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:26.992+0000 7f0c32f93700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c2c100ed0 0x7f0c2c1992e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:31:26.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:26.992+0000 7f0c32f93700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0c2c101870 0x7f0c2c1a0360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:31:26.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:26.992+0000 7f0c32f93700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2c1a0930 con 0x7f0c2c0ffcd0 2024-01-24T13:31:26.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:26.992+0000 7f0c32f93700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2c1a0a70 con 0x7f0c2c101870 2024-01-24T13:31:26.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:26.992+0000 7f0c32f93700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2c1a0c00 con 0x7f0c2c100ed0 2024-01-24T13:31:26.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:26.992+0000 7f0c30d2f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c2c0ffcd0 0x7f0c2c198da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:31:26.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:26.992+0000 7f0c2bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c2c100ed0 0x7f0c2c1992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:31:26.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:26.992+0000 7f0c30d2f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c2c0ffcd0 0x7f0c2c198da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55252/0 (socket says 172.21.15.29:55252) 2024-01-24T13:31:26.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:26.992+0000 7f0c30d2f700 1 -- 172.21.15.29:0/783803404 learned_addr learned my addr 172.21.15.29:0/783803404 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:31:26.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:26.993+0000 7f0c31530700 1 --2- 172.21.15.29:0/783803404 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0c2c101870 0x7f0c2c1a0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:31:26.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:26.993+0000 7f0c30d2f700 1 -- 172.21.15.29:0/783803404 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c2c100ed0 msgr2=0x7f0c2c1992e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:31:26.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:26.993+0000 7f0c30d2f700 1 --2- 172.21.15.29:0/783803404 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c2c100ed0 0x7f0c2c1992e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:26.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:26.993+0000 7f0c30d2f700 1 -- 172.21.15.29:0/783803404 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0c2c101870 msgr2=0x7f0c2c1a0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:31:26.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:26.993+0000 7f0c30d2f700 1 --2- 172.21.15.29:0/783803404 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0c2c101870 0x7f0c2c1a0360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:26.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:26.993+0000 7f0c30d2f700 1 -- 172.21.15.29:0/783803404 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c24009d00 con 0x7f0c2c0ffcd0 2024-01-24T13:31:26.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:26.993+0000 7f0c30d2f700 1 --2- 172.21.15.29:0/783803404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c2c0ffcd0 0x7f0c2c198da0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c00d9b0 tx=0x7f0c1c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:31:26.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:26.993+0000 7f0c29ffb700 1 -- 172.21.15.29:0/783803404 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c1c00b950 con 0x7f0c2c0ffcd0 2024-01-24T13:31:26.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:26.993+0000 7f0c29ffb700 1 -- 172.21.15.29:0/783803404 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0c1c007620 con 0x7f0c2c0ffcd0 2024-01-24T13:31:26.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:26.993+0000 7f0c32f93700 1 -- 172.21.15.29:0/783803404 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c2c1a0ee0 con 0x7f0c2c0ffcd0 2024-01-24T13:31:26.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:26.993+0000 7f0c29ffb700 1 -- 172.21.15.29:0/783803404 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c1c00f3c0 con 0x7f0c2c0ffcd0 2024-01-24T13:31:27.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:26.993+0000 7f0c32f93700 1 -- 172.21.15.29:0/783803404 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c2c1a1300 con 0x7f0c2c0ffcd0 2024-01-24T13:31:27.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:26.994+0000 7f0c29ffb700 1 -- 172.21.15.29:0/783803404 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 45324+0+0 (secure 0 0 0) 0x7f0c1c00f520 con 0x7f0c2c0ffcd0 2024-01-24T13:31:27.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:26.994+0000 7f0c29ffb700 1 --2- 172.21.15.29:0/783803404 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f0c140385d0 0x7f0c1403aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:31:27.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:26.994+0000 7f0c29ffb700 1 -- 172.21.15.29:0/783803404 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0c1c04ab80 con 0x7f0c2c0ffcd0 2024-01-24T13:31:27.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:26.994+0000 7f0c2bfff700 1 --2- 172.21.15.29:0/783803404 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f0c140385d0 0x7f0c1403aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:31:27.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:26.995+0000 7f0c32f93700 1 -- 172.21.15.29:0/783803404 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c2c10b6c0 con 0x7f0c2c0ffcd0 2024-01-24T13:31:27.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:26.995+0000 7f0c2bfff700 1 --2- 172.21.15.29:0/783803404 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f0c140385d0 0x7f0c1403aa90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0c20007b60 tx=0x7f0c200073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:31:27.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:27.000+0000 7f0c29ffb700 1 -- 172.21.15.29:0/783803404 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f0c1c0228f0 con 0x7f0c2c0ffcd0 2024-01-24T13:31:27.054 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jan 24 13:31:26 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-z[46787]: 2024-01-24T13:31:26.705+0000 7f42ed0da200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-24T13:31:27.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:27.223+0000 7f0c32f93700 1 -- 172.21.15.29:0/783803404 --> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f0c2c069820 con 0x7f0c140385d0 2024-01-24T13:31:27.232 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Jan 24 13:31:26 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-y[46456]: 2024-01-24T13:31:26.980+0000 7f424811c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-24T13:31:27.701 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jan 24 13:31:27 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-z[46787]: 2024-01-24T13:31:27.331+0000 7f42ed0da200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-24T13:31:27.701 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jan 24 13:31:27 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-z[46787]: 2024-01-24T13:31:27.510+0000 7f42ed0da200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-24T13:31:27.732 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Jan 24 13:31:27 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-y[46456]: 2024-01-24T13:31:27.266+0000 7f424811c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-24T13:31:27.732 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Jan 24 13:31:27 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-y[46456]: 2024-01-24T13:31:27.357+0000 7f424811c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-24T13:31:28.054 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jan 24 13:31:27 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-z[46787]: 2024-01-24T13:31:27.700+0000 7f42ed0da200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-24T13:31:28.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:27 smithi187 ceph-mon[44911]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:28.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:27 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:28.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:27 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:28.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:27 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:31:28.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:27 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:28.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:27 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:31:28.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:27 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:28.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:27 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T13:31:28.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:27 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T13:31:28.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:27 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:28.217 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Jan 24 13:31:28 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-y[46456]: 2024-01-24T13:31:28.121+0000 7f424811c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-24T13:31:28.217 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:27 smithi163 ceph-mon[44969]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:28.217 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:27 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:28.217 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:27 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:28.217 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:27 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:31:28.217 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:27 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:28.217 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:27 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:31:28.218 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:27 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:28.218 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:27 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T13:31:28.218 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:27 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T13:31:28.218 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:27 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:28.242 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:27 smithi029 ceph-mon[40651]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:28.242 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:27 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:28.242 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:27 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:28.242 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:27 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:31:28.242 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:27 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:28.242 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:27 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:31:28.242 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:27 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:28.242 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:27 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T13:31:28.243 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:27 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T13:31:28.243 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:27 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:28.482 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Jan 24 13:31:28 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-y[46456]: 2024-01-24T13:31:28.216+0000 7f424811c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-24T13:31:28.482 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Jan 24 13:31:28 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-y[46456]: 2024-01-24T13:31:28.362+0000 7f424811c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-24T13:31:28.874 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jan 24 13:31:28 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-z[46787]: 2024-01-24T13:31:28.613+0000 7f42ed0da200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-24T13:31:29.153 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:28 smithi187 ceph-mon[44911]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:31:29.153 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jan 24 13:31:28 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-z[46787]: 2024-01-24T13:31:28.878+0000 7f42ed0da200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-24T13:31:29.153 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jan 24 13:31:28 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-z[46787]: 2024-01-24T13:31:28.965+0000 7f42ed0da200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-24T13:31:29.171 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:28 smithi163 ceph-mon[44969]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:31:29.171 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Jan 24 13:31:28 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-y[46456]: 2024-01-24T13:31:28.989+0000 7f424811c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-24T13:31:29.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:28 smithi029 ceph-mon[40651]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:31:29.482 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Jan 24 13:31:29 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-y[46456]: 2024-01-24T13:31:29.170+0000 7f424811c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-24T13:31:29.482 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Jan 24 13:31:29 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-y[46456]: 2024-01-24T13:31:29.362+0000 7f424811c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-24T13:31:29.554 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jan 24 13:31:29 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-z[46787]: 2024-01-24T13:31:29.152+0000 7f42ed0da200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-24T13:31:30.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:30.027+0000 7f0c29ffb700 1 -- 172.21.15.29:0/783803404 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 14) v1 ==== 90169+0+0 (secure 0 0 0) 0x7f0c1c01b4f0 con 0x7f0c2c0ffcd0 2024-01-24T13:31:30.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:29 smithi163 ceph-mon[44969]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:30.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:29 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/762004206' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9857a607-2e6f-41a5-a86a-795970be6210"}]: dispatch 2024-01-24T13:31:30.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:29 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/762004206' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9857a607-2e6f-41a5-a86a-795970be6210"}]': finished 2024-01-24T13:31:30.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:29 smithi163 ceph-mon[44969]: osdmap e5: 1 total, 0 up, 1 in 2024-01-24T13:31:30.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:29 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-24T13:31:30.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:29 smithi163 ceph-mon[44969]: Standby manager daemon z started 2024-01-24T13:31:30.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:29 smithi163 ceph-mon[44969]: from='mgr.? 172.21.15.187:0/2974847774' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-01-24T13:31:30.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:29 smithi163 ceph-mon[44969]: from='mgr.? 172.21.15.187:0/2974847774' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-01-24T13:31:30.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:29 smithi163 ceph-mon[44969]: from='mgr.? 172.21.15.187:0/2974847774' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-01-24T13:31:30.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:29 smithi163 ceph-mon[44969]: from='mgr.? 172.21.15.187:0/2974847774' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-01-24T13:31:30.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:29 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/2804591202' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T13:31:30.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:29 smithi187 ceph-mon[44911]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:30.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:29 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/762004206' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9857a607-2e6f-41a5-a86a-795970be6210"}]: dispatch 2024-01-24T13:31:30.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:29 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/762004206' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9857a607-2e6f-41a5-a86a-795970be6210"}]': finished 2024-01-24T13:31:30.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:29 smithi187 ceph-mon[44911]: osdmap e5: 1 total, 0 up, 1 in 2024-01-24T13:31:30.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:29 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-24T13:31:30.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:29 smithi187 ceph-mon[44911]: Standby manager daemon z started 2024-01-24T13:31:30.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:29 smithi187 ceph-mon[44911]: from='mgr.? 172.21.15.187:0/2974847774' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-01-24T13:31:30.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:29 smithi187 ceph-mon[44911]: from='mgr.? 172.21.15.187:0/2974847774' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-01-24T13:31:30.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:29 smithi187 ceph-mon[44911]: from='mgr.? 172.21.15.187:0/2974847774' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-01-24T13:31:30.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:29 smithi187 ceph-mon[44911]: from='mgr.? 172.21.15.187:0/2974847774' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-01-24T13:31:30.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:29 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/2804591202' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T13:31:30.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:29 smithi029 ceph-mon[40651]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:30.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:29 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/762004206' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9857a607-2e6f-41a5-a86a-795970be6210"}]: dispatch 2024-01-24T13:31:30.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:29 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/762004206' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9857a607-2e6f-41a5-a86a-795970be6210"}]': finished 2024-01-24T13:31:30.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:29 smithi029 ceph-mon[40651]: osdmap e5: 1 total, 0 up, 1 in 2024-01-24T13:31:30.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:29 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-24T13:31:30.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:29 smithi029 ceph-mon[40651]: Standby manager daemon z started 2024-01-24T13:31:30.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:29 smithi029 ceph-mon[40651]: from='mgr.? 172.21.15.187:0/2974847774' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-01-24T13:31:30.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:29 smithi029 ceph-mon[40651]: from='mgr.? 172.21.15.187:0/2974847774' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-01-24T13:31:30.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:29 smithi029 ceph-mon[40651]: from='mgr.? 172.21.15.187:0/2974847774' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-01-24T13:31:30.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:29 smithi029 ceph-mon[40651]: from='mgr.? 172.21.15.187:0/2974847774' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-01-24T13:31:30.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:29 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/2804591202' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T13:31:30.549 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Jan 24 13:31:30 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-y[46456]: 2024-01-24T13:31:30.282+0000 7f424811c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-24T13:31:30.828 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Jan 24 13:31:30 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-y[46456]: 2024-01-24T13:31:30.548+0000 7f424811c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-24T13:31:30.828 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Jan 24 13:31:30 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-y[46456]: 2024-01-24T13:31:30.634+0000 7f424811c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-24T13:31:31.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:31.035+0000 7f0c29ffb700 1 -- 172.21.15.29:0/783803404 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 15) v1 ==== 135014+0+0 (secure 0 0 0) 0x7f0c1c01a3b0 con 0x7f0c2c0ffcd0 2024-01-24T13:31:31.232 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Jan 24 13:31:30 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-y[46456]: 2024-01-24T13:31:30.827+0000 7f424811c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-24T13:31:31.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:31 smithi163 ceph-mon[44969]: mgrmap e14: x(active, since 59s), standbys: z 2024-01-24T13:31:31.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:31 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-01-24T13:31:31.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:31 smithi163 ceph-mon[44969]: Standby manager daemon y started 2024-01-24T13:31:31.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:31 smithi163 ceph-mon[44969]: from='mgr.? 172.21.15.163:0/3762734607' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-01-24T13:31:31.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:31 smithi163 ceph-mon[44969]: from='mgr.? 172.21.15.163:0/3762734607' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-01-24T13:31:31.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:31 smithi163 ceph-mon[44969]: from='mgr.? 172.21.15.163:0/3762734607' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-01-24T13:31:31.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:31 smithi163 ceph-mon[44969]: from='mgr.? 172.21.15.163:0/3762734607' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-01-24T13:31:31.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:31 smithi187 ceph-mon[44911]: mgrmap e14: x(active, since 59s), standbys: z 2024-01-24T13:31:31.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:31 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-01-24T13:31:31.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:31 smithi187 ceph-mon[44911]: Standby manager daemon y started 2024-01-24T13:31:31.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:31 smithi187 ceph-mon[44911]: from='mgr.? 172.21.15.163:0/3762734607' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-01-24T13:31:31.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:31 smithi187 ceph-mon[44911]: from='mgr.? 172.21.15.163:0/3762734607' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-01-24T13:31:31.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:31 smithi187 ceph-mon[44911]: from='mgr.? 172.21.15.163:0/3762734607' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-01-24T13:31:31.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:31 smithi187 ceph-mon[44911]: from='mgr.? 172.21.15.163:0/3762734607' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-01-24T13:31:31.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:31 smithi029 ceph-mon[40651]: mgrmap e14: x(active, since 59s), standbys: z 2024-01-24T13:31:31.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:31 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-01-24T13:31:31.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:31 smithi029 ceph-mon[40651]: Standby manager daemon y started 2024-01-24T13:31:31.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:31 smithi029 ceph-mon[40651]: from='mgr.? 172.21.15.163:0/3762734607' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-01-24T13:31:31.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:31 smithi029 ceph-mon[40651]: from='mgr.? 172.21.15.163:0/3762734607' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-01-24T13:31:31.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:31 smithi029 ceph-mon[40651]: from='mgr.? 172.21.15.163:0/3762734607' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-01-24T13:31:31.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:31 smithi029 ceph-mon[40651]: from='mgr.? 172.21.15.163:0/3762734607' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-01-24T13:31:32.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:32 smithi187 ceph-mon[44911]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:32.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:32 smithi187 ceph-mon[44911]: mgrmap e15: x(active, since 60s), standbys: z, y 2024-01-24T13:31:32.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:32 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-01-24T13:31:32.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:32 smithi029 ceph-mon[40651]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:32.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:32 smithi029 ceph-mon[40651]: mgrmap e15: x(active, since 60s), standbys: z, y 2024-01-24T13:31:32.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:32 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-01-24T13:31:32.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:32 smithi163 ceph-mon[44969]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:32.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:32 smithi163 ceph-mon[44969]: mgrmap e15: x(active, since 60s), standbys: z, y 2024-01-24T13:31:32.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:32 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-01-24T13:31:34.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:34 smithi029 ceph-mon[40651]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:34.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:34 smithi187 ceph-mon[44911]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:34.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:34 smithi163 ceph-mon[44969]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:35.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:35 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-01-24T13:31:35.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:35 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:35.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:35 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-01-24T13:31:35.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:35 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:35.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:35 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-01-24T13:31:35.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:35 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:36.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:36 smithi187 ceph-mon[44911]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:36.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:36 smithi187 ceph-mon[44911]: Deploying daemon osd.0 on smithi029 2024-01-24T13:31:36.345 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:36 smithi029 ceph-mon[40651]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:36.345 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:36 smithi029 ceph-mon[40651]: Deploying daemon osd.0 on smithi029 2024-01-24T13:31:36.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:36 smithi163 ceph-mon[44969]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:36.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:36 smithi163 ceph-mon[44969]: Deploying daemon osd.0 on smithi029 2024-01-24T13:31:38.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:38 smithi187 ceph-mon[44911]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:38.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:38 smithi029 ceph-mon[40651]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:38.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:38 smithi163 ceph-mon[44969]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:39.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:39 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:39.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:39 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:39.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:39 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:31:39.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:39 smithi029 ceph-mon[40651]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:39.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:39 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:39.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:39 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:39.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:39 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:31:39.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:39 smithi163 ceph-mon[44969]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:40.054 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:39 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:40.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:39 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:40.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:39 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:31:40.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:39 smithi187 ceph-mon[44911]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:40.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:40.869+0000 7f0c29ffb700 1 -- 172.21.15.29:0/783803404 <== mgr.14152 v2:172.21.15.29:6800/2498991338 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f0c2c069820 con 0x7f0c140385d0 2024-01-24T13:31:40.871 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 0 on host 'smithi029' 2024-01-24T13:31:40.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:40.872+0000 7f0c32f93700 1 -- 172.21.15.29:0/783803404 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f0c140385d0 msgr2=0x7f0c1403aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:31:40.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:40.872+0000 7f0c32f93700 1 --2- 172.21.15.29:0/783803404 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f0c140385d0 0x7f0c1403aa90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0c20007b60 tx=0x7f0c200073d0 comp rx=0 tx=0).stop 2024-01-24T13:31:40.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:40.872+0000 7f0c32f93700 1 -- 172.21.15.29:0/783803404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c2c0ffcd0 msgr2=0x7f0c2c198da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:31:40.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:40.872+0000 7f0c32f93700 1 --2- 172.21.15.29:0/783803404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c2c0ffcd0 0x7f0c2c198da0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c00d9b0 tx=0x7f0c1c00dcc0 comp rx=0 tx=0).stop 2024-01-24T13:31:40.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:40.872+0000 7f0c32f93700 1 -- 172.21.15.29:0/783803404 shutdown_connections 2024-01-24T13:31:40.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:40.872+0000 7f0c32f93700 1 --2- 172.21.15.29:0/783803404 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0c2c101870 0x7f0c2c1a0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:40.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:40.872+0000 7f0c32f93700 1 --2- 172.21.15.29:0/783803404 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f0c140385d0 0x7f0c1403aa90 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:40.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:40.872+0000 7f0c32f93700 1 --2- 172.21.15.29:0/783803404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c2c0ffcd0 0x7f0c2c198da0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:40.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:40.872+0000 7f0c32f93700 1 --2- 172.21.15.29:0/783803404 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c2c100ed0 0x7f0c2c1992e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:40.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:40.872+0000 7f0c32f93700 1 -- 172.21.15.29:0/783803404 >> 172.21.15.29:0/783803404 conn(0x7f0c2c0fb250 msgr2=0x7f0c2c069110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:31:40.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:40.873+0000 7f0c32f93700 1 -- 172.21.15.29:0/783803404 shutdown_connections 2024-01-24T13:31:40.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:31:40.873+0000 7f0c32f93700 1 -- 172.21.15.29:0/783803404 wait complete. 2024-01-24T13:31:41.046 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:40 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:41.046 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:40 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:41.046 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:40 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:41.046 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:40 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:31:41.046 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:40 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:41.046 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:40 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:41.046 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:40 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:41.046 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:40 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:31:41.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:40 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:41.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:40 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:41.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:40 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:41.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:40 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:31:41.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:40 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:41.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:40 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:41.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:40 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:41.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:40 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:31:41.262 DEBUG:teuthology.orchestra.run.smithi029:osd.0> sudo journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@osd.0.service 2024-01-24T13:31:41.264 INFO:tasks.cephadm:Deploying osd.1 on smithi163 with /dev/vg_nvme/lv_4... 2024-01-24T13:31:41.264 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-01-24T13:31:41.300 INFO:journalctl@ceph.osd.0.smithi029.stdout:-- Logs begin at Wed 2024-01-24 13:13:42 UTC. -- 2024-01-24T13:31:41.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:40 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:41.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:40 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:41.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:40 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:41.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:40 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:31:41.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:40 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:41.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:40 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:41.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:40 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:41.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:40 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:31:41.492 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.b/config 2024-01-24T13:31:42.075 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:41 smithi029 ceph-mon[40651]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:42.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:41 smithi163 ceph-mon[44969]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:42.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:41 smithi187 ceph-mon[44911]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:42.761 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-01-24T13:31:42.761 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-01-24T13:31:42.761 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10+0 records in 2024-01-24T13:31:42.761 INFO:teuthology.orchestra.run.smithi163.stderr:10+0 records out 2024-01-24T13:31:42.761 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113067 s, 927 MB/s 2024-01-24T13:31:42.761 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping successful for: 2024-01-24T13:31:42.849 INFO:journalctl@ceph.osd.0.smithi029.stdout:Jan 24 13:31:42 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-0[59238]: 2024-01-24T13:31:42.550+0000 7f606f055640 -1 osd.0 0 log_to_monitors true 2024-01-24T13:31:43.124 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:42 smithi163 ceph-mon[44969]: from='osd.0 [v2:172.21.15.29:6802/2406629076,v1:172.21.15.29:6803/2406629076]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-01-24T13:31:43.125 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:42 smithi163 ceph-mon[44969]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-01-24T13:31:43.126 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph orch daemon add osd smithi163:vg_nvme/lv_4 2024-01-24T13:31:43.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:42 smithi029 ceph-mon[40651]: from='osd.0 [v2:172.21.15.29:6802/2406629076,v1:172.21.15.29:6803/2406629076]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-01-24T13:31:43.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:42 smithi029 ceph-mon[40651]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-01-24T13:31:43.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:42 smithi187 ceph-mon[44911]: from='osd.0 [v2:172.21.15.29:6802/2406629076,v1:172.21.15.29:6803/2406629076]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-01-24T13:31:43.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:42 smithi187 ceph-mon[44911]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-01-24T13:31:43.340 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.b/config 2024-01-24T13:31:44.131 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:43 smithi163 ceph-mon[44969]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:44.131 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:43 smithi163 ceph-mon[44969]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-01-24T13:31:44.131 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:43 smithi163 ceph-mon[44969]: osdmap e6: 1 total, 0 up, 1 in 2024-01-24T13:31:44.131 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:43 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-24T13:31:44.131 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:43 smithi163 ceph-mon[44969]: from='osd.0 [v2:172.21.15.29:6802/2406629076,v1:172.21.15.29:6803/2406629076]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-01-24T13:31:44.131 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:43 smithi163 ceph-mon[44969]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-01-24T13:31:44.132 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:43 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:44.132 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:43 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:44.132 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:43 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:31:44.132 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:43 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:44.132 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:43 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:44.132 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:43 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:31:44.132 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:43 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:44.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.301+0000 7fde8bf5c700 1 -- 172.21.15.163:0/2586097481 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde84100ed0 msgr2=0x7fde84101330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:31:44.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.301+0000 7fde8bf5c700 1 --2- 172.21.15.163:0/2586097481 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde84100ed0 0x7fde84101330 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fde74009990 tx=0x7fde74009ca0 comp rx=0 tx=0).stop 2024-01-24T13:31:44.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.301+0000 7fde8bf5c700 1 -- 172.21.15.163:0/2586097481 shutdown_connections 2024-01-24T13:31:44.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.301+0000 7fde8bf5c700 1 --2- 172.21.15.163:0/2586097481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde84101870 0x7fde84108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:44.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.301+0000 7fde8bf5c700 1 --2- 172.21.15.163:0/2586097481 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fde840ffcd0 0x7fde841000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:44.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.301+0000 7fde8bf5c700 1 --2- 172.21.15.163:0/2586097481 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde84100ed0 0x7fde84101330 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:44.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.301+0000 7fde8bf5c700 1 -- 172.21.15.163:0/2586097481 >> 172.21.15.163:0/2586097481 conn(0x7fde840fb250 msgr2=0x7fde840fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:31:44.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.301+0000 7fde8bf5c700 1 -- 172.21.15.163:0/2586097481 shutdown_connections 2024-01-24T13:31:44.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.301+0000 7fde8bf5c700 1 -- 172.21.15.163:0/2586097481 wait complete. 2024-01-24T13:31:44.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.301+0000 7fde8bf5c700 1 Processor -- start 2024-01-24T13:31:44.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.302+0000 7fde8bf5c700 1 -- start start 2024-01-24T13:31:44.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.302+0000 7fde8bf5c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde840ffcd0 0x7fde84196aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:31:44.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.302+0000 7fde8bf5c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde84100ed0 0x7fde84196fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:31:44.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.302+0000 7fde8bf5c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fde84101870 0x7fde8419e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:31:44.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.302+0000 7fde8bf5c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8419e630 con 0x7fde840ffcd0 2024-01-24T13:31:44.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.302+0000 7fde8bf5c700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8419e770 con 0x7fde84101870 2024-01-24T13:31:44.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.302+0000 7fde8bf5c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8419e900 con 0x7fde84100ed0 2024-01-24T13:31:44.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.302+0000 7fde8a4f9700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fde84101870 0x7fde8419e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:31:44.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.302+0000 7fde8a4f9700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fde84101870 0x7fde8419e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:38304/0 (socket says 172.21.15.163:38304) 2024-01-24T13:31:44.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.302+0000 7fde894f7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde84100ed0 0x7fde84196fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:31:44.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.302+0000 7fde8a4f9700 1 -- 172.21.15.163:0/2200502853 learned_addr learned my addr 172.21.15.163:0/2200502853 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-01-24T13:31:44.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.302+0000 7fde89cf8700 1 --2- 172.21.15.163:0/2200502853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde840ffcd0 0x7fde84196aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:31:44.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.302+0000 7fde8a4f9700 1 -- 172.21.15.163:0/2200502853 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde84100ed0 msgr2=0x7fde84196fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:31:44.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.302+0000 7fde8a4f9700 1 --2- 172.21.15.163:0/2200502853 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde84100ed0 0x7fde84196fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:44.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.302+0000 7fde8a4f9700 1 -- 172.21.15.163:0/2200502853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde840ffcd0 msgr2=0x7fde84196aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:31:44.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.302+0000 7fde8a4f9700 1 --2- 172.21.15.163:0/2200502853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde840ffcd0 0x7fde84196aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:44.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.302+0000 7fde8a4f9700 1 -- 172.21.15.163:0/2200502853 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde740095f0 con 0x7fde84101870 2024-01-24T13:31:44.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.302+0000 7fde8a4f9700 1 --2- 172.21.15.163:0/2200502853 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fde84101870 0x7fde8419e060 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fde7c007ee0 tx=0x7fde7c00d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:31:44.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:43 smithi187 ceph-mon[44911]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:44.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:43 smithi187 ceph-mon[44911]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-01-24T13:31:44.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:43 smithi187 ceph-mon[44911]: osdmap e6: 1 total, 0 up, 1 in 2024-01-24T13:31:44.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:43 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-24T13:31:44.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:43 smithi187 ceph-mon[44911]: from='osd.0 [v2:172.21.15.29:6802/2406629076,v1:172.21.15.29:6803/2406629076]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-01-24T13:31:44.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:43 smithi187 ceph-mon[44911]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-01-24T13:31:44.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:43 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:44.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:43 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:44.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:43 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:31:44.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:43 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:44.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:43 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:44.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:43 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:31:44.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:43 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:44.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.302+0000 7fde7affd700 1 -- 172.21.15.163:0/2200502853 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7c017070 con 0x7fde84101870 2024-01-24T13:31:44.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.302+0000 7fde8bf5c700 1 -- 172.21.15.163:0/2200502853 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde8419ebe0 con 0x7fde84101870 2024-01-24T13:31:44.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.302+0000 7fde7affd700 1 -- 172.21.15.163:0/2200502853 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fde7c00f040 con 0x7fde84101870 2024-01-24T13:31:44.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.303+0000 7fde7affd700 1 -- 172.21.15.163:0/2200502853 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7c005610 con 0x7fde84101870 2024-01-24T13:31:44.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.303+0000 7fde8bf5c700 1 -- 172.21.15.163:0/2200502853 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde8419f090 con 0x7fde84101870 2024-01-24T13:31:44.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.303+0000 7fde8bf5c700 1 -- 172.21.15.163:0/2200502853 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde8410b4d0 con 0x7fde84101870 2024-01-24T13:31:44.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.304+0000 7fde7affd700 1 -- 172.21.15.163:0/2200502853 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 15) v1 ==== 135014+0+0 (secure 0 0 0) 0x7fde7c004c40 con 0x7fde84101870 2024-01-24T13:31:44.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.305+0000 7fde7affd700 1 --2- 172.21.15.163:0/2200502853 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fde700a0490 0x7fde700a2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:31:44.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.305+0000 7fde7affd700 1 -- 172.21.15.163:0/2200502853 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1487+0+0 (secure 0 0 0) 0x7fde7c0c94a0 con 0x7fde84101870 2024-01-24T13:31:44.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.305+0000 7fde89cf8700 1 --2- 172.21.15.163:0/2200502853 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fde700a0490 0x7fde700a2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:31:44.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.306+0000 7fde89cf8700 1 --2- 172.21.15.163:0/2200502853 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fde700a0490 0x7fde700a2950 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fde84100d30 tx=0x7fde80009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:31:44.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.309+0000 7fde7affd700 1 -- 172.21.15.163:0/2200502853 <== mon.2 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fde7c094bd0 con 0x7fde84101870 2024-01-24T13:31:44.349 INFO:journalctl@ceph.osd.0.smithi029.stdout:Jan 24 13:31:43 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-0[59238]: 2024-01-24T13:31:43.888+0000 7f60646ba700 -1 osd.0 0 waiting for initial osdmap 2024-01-24T13:31:44.349 INFO:journalctl@ceph.osd.0.smithi029.stdout:Jan 24 13:31:43 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-0[59238]: 2024-01-24T13:31:43.892+0000 7f60604af700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-24T13:31:44.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:43 smithi029 ceph-mon[40651]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:44.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:43 smithi029 ceph-mon[40651]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-01-24T13:31:44.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:43 smithi029 ceph-mon[40651]: osdmap e6: 1 total, 0 up, 1 in 2024-01-24T13:31:44.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:43 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-24T13:31:44.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:43 smithi029 ceph-mon[40651]: from='osd.0 [v2:172.21.15.29:6802/2406629076,v1:172.21.15.29:6803/2406629076]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-01-24T13:31:44.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:43 smithi029 ceph-mon[40651]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-01-24T13:31:44.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:43 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:44.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:43 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:44.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:43 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:31:44.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:43 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:44.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:43 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:44.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:43 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:31:44.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:43 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:44.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:44.505+0000 7fde8bf5c700 1 -- 172.21.15.163:0/2200502853 --> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fde84068f90 con 0x7fde700a0490 2024-01-24T13:31:45.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:44 smithi163 ceph-mon[44969]: Detected new or changed devices on smithi029 2024-01-24T13:31:45.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:44 smithi163 ceph-mon[44969]: Adjusting osd_memory_target on smithi029 to 17118M 2024-01-24T13:31:45.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:44 smithi163 ceph-mon[44969]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-01-24T13:31:45.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:44 smithi163 ceph-mon[44969]: osdmap e7: 1 total, 0 up, 1 in 2024-01-24T13:31:45.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:44 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-24T13:31:45.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:44 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-24T13:31:45.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:44 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T13:31:45.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:44 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T13:31:45.222 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:44 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:45.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:44 smithi187 ceph-mon[44911]: Detected new or changed devices on smithi029 2024-01-24T13:31:45.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:44 smithi187 ceph-mon[44911]: Adjusting osd_memory_target on smithi029 to 17118M 2024-01-24T13:31:45.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:44 smithi187 ceph-mon[44911]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-01-24T13:31:45.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:44 smithi187 ceph-mon[44911]: osdmap e7: 1 total, 0 up, 1 in 2024-01-24T13:31:45.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:44 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-24T13:31:45.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:44 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-24T13:31:45.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:44 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T13:31:45.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:44 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T13:31:45.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:44 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:45.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:44 smithi029 ceph-mon[40651]: Detected new or changed devices on smithi029 2024-01-24T13:31:45.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:44 smithi029 ceph-mon[40651]: Adjusting osd_memory_target on smithi029 to 17118M 2024-01-24T13:31:45.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:44 smithi029 ceph-mon[40651]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-01-24T13:31:45.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:44 smithi029 ceph-mon[40651]: osdmap e7: 1 total, 0 up, 1 in 2024-01-24T13:31:45.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:44 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-24T13:31:45.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:44 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-24T13:31:45.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:44 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T13:31:45.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:44 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T13:31:45.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:44 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:46.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:45 smithi163 ceph-mon[44969]: purged_snaps scrub starts 2024-01-24T13:31:46.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:45 smithi163 ceph-mon[44969]: purged_snaps scrub ok 2024-01-24T13:31:46.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:45 smithi163 ceph-mon[44969]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:31:46.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:45 smithi163 ceph-mon[44969]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:46.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:45 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-24T13:31:46.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:45 smithi163 ceph-mon[44969]: osd.0 [v2:172.21.15.29:6802/2406629076,v1:172.21.15.29:6803/2406629076] boot 2024-01-24T13:31:46.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:45 smithi163 ceph-mon[44969]: osdmap e8: 1 total, 1 up, 1 in 2024-01-24T13:31:46.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:45 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-24T13:31:46.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:45 smithi187 ceph-mon[44911]: purged_snaps scrub starts 2024-01-24T13:31:46.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:45 smithi187 ceph-mon[44911]: purged_snaps scrub ok 2024-01-24T13:31:46.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:45 smithi187 ceph-mon[44911]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:31:46.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:45 smithi187 ceph-mon[44911]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:46.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:45 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-24T13:31:46.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:45 smithi187 ceph-mon[44911]: osd.0 [v2:172.21.15.29:6802/2406629076,v1:172.21.15.29:6803/2406629076] boot 2024-01-24T13:31:46.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:45 smithi187 ceph-mon[44911]: osdmap e8: 1 total, 1 up, 1 in 2024-01-24T13:31:46.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:45 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-24T13:31:46.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:45 smithi029 ceph-mon[40651]: purged_snaps scrub starts 2024-01-24T13:31:46.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:45 smithi029 ceph-mon[40651]: purged_snaps scrub ok 2024-01-24T13:31:46.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:45 smithi029 ceph-mon[40651]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:31:46.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:45 smithi029 ceph-mon[40651]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T13:31:46.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:45 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-24T13:31:46.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:45 smithi029 ceph-mon[40651]: osd.0 [v2:172.21.15.29:6802/2406629076,v1:172.21.15.29:6803/2406629076] boot 2024-01-24T13:31:46.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:45 smithi029 ceph-mon[40651]: osdmap e8: 1 total, 1 up, 1 in 2024-01-24T13:31:46.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:45 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-24T13:31:47.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:47 smithi187 ceph-mon[44911]: osdmap e9: 1 total, 1 up, 1 in 2024-01-24T13:31:47.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:47 smithi187 ceph-mon[44911]: from='client.? 172.21.15.163:0/4105293987' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8bb2ba48-23d4-4bbd-a912-ae74f9e7f6b0"}]: dispatch 2024-01-24T13:31:47.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:47 smithi187 ceph-mon[44911]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8bb2ba48-23d4-4bbd-a912-ae74f9e7f6b0"}]: dispatch 2024-01-24T13:31:47.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:47 smithi187 ceph-mon[44911]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8bb2ba48-23d4-4bbd-a912-ae74f9e7f6b0"}]': finished 2024-01-24T13:31:47.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:47 smithi187 ceph-mon[44911]: osdmap e10: 2 total, 1 up, 2 in 2024-01-24T13:31:47.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:47 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-24T13:31:47.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:47 smithi187 ceph-mon[44911]: from='client.? 172.21.15.163:0/2634275626' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T13:31:47.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:47 smithi029 ceph-mon[40651]: osdmap e9: 1 total, 1 up, 1 in 2024-01-24T13:31:47.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:47 smithi029 ceph-mon[40651]: from='client.? 172.21.15.163:0/4105293987' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8bb2ba48-23d4-4bbd-a912-ae74f9e7f6b0"}]: dispatch 2024-01-24T13:31:47.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:47 smithi029 ceph-mon[40651]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8bb2ba48-23d4-4bbd-a912-ae74f9e7f6b0"}]: dispatch 2024-01-24T13:31:47.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:47 smithi029 ceph-mon[40651]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8bb2ba48-23d4-4bbd-a912-ae74f9e7f6b0"}]': finished 2024-01-24T13:31:47.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:47 smithi029 ceph-mon[40651]: osdmap e10: 2 total, 1 up, 2 in 2024-01-24T13:31:47.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:47 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-24T13:31:47.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:47 smithi029 ceph-mon[40651]: from='client.? 172.21.15.163:0/2634275626' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T13:31:47.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:47 smithi163 ceph-mon[44969]: osdmap e9: 1 total, 1 up, 1 in 2024-01-24T13:31:47.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:47 smithi163 ceph-mon[44969]: from='client.? 172.21.15.163:0/4105293987' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8bb2ba48-23d4-4bbd-a912-ae74f9e7f6b0"}]: dispatch 2024-01-24T13:31:47.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:47 smithi163 ceph-mon[44969]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8bb2ba48-23d4-4bbd-a912-ae74f9e7f6b0"}]: dispatch 2024-01-24T13:31:47.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:47 smithi163 ceph-mon[44969]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8bb2ba48-23d4-4bbd-a912-ae74f9e7f6b0"}]': finished 2024-01-24T13:31:47.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:47 smithi163 ceph-mon[44969]: osdmap e10: 2 total, 1 up, 2 in 2024-01-24T13:31:47.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:47 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-24T13:31:47.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:47 smithi163 ceph-mon[44969]: from='client.? 172.21.15.163:0/2634275626' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T13:31:48.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:48 smithi187 ceph-mon[44911]: pgmap v38: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T13:31:48.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:48 smithi029 ceph-mon[40651]: pgmap v38: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T13:31:48.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:48 smithi163 ceph-mon[44969]: pgmap v38: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T13:31:50.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:50 smithi187 ceph-mon[44911]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T13:31:50.324 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:50 smithi163 ceph-mon[44969]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T13:31:50.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:50 smithi029 ceph-mon[40651]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T13:31:52.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:52 smithi163 ceph-mon[44969]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T13:31:52.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:52 smithi187 ceph-mon[44911]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T13:31:52.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:52 smithi029 ceph-mon[40651]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T13:31:53.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:53 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-01-24T13:31:53.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:53 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:53.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:53 smithi163 ceph-mon[44969]: Deploying daemon osd.1 on smithi163 2024-01-24T13:31:53.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:53 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-01-24T13:31:53.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:53 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:53.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:53 smithi187 ceph-mon[44911]: Deploying daemon osd.1 on smithi163 2024-01-24T13:31:53.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:53 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-01-24T13:31:53.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:53 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:53.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:53 smithi029 ceph-mon[40651]: Deploying daemon osd.1 on smithi163 2024-01-24T13:31:54.210 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:54 smithi163 ceph-mon[44969]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T13:31:54.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:54 smithi187 ceph-mon[44911]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T13:31:54.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:54 smithi029 ceph-mon[40651]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T13:31:56.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:56 smithi187 ceph-mon[44911]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T13:31:56.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:56 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:56.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:56 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:31:56.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:56 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:56.310 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:56 smithi163 ceph-mon[44969]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T13:31:56.310 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:56 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:56.310 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:56 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:31:56.310 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:56 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:56.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:56 smithi029 ceph-mon[40651]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T13:31:56.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:56 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:56.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:56 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:31:56.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:56 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:57.940 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:57 smithi163 ceph-mon[44969]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T13:31:57.940 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:57 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:57.941 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:57 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:57.941 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:57 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:57.941 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:57 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:31:57.941 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:57 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:58.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:58.020+0000 7fde7affd700 1 -- 172.21.15.163:0/2200502853 <== mgr.14152 v2:172.21.15.29:6800/2498991338 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fde84068f90 con 0x7fde700a0490 2024-01-24T13:31:58.021 INFO:teuthology.orchestra.run.smithi163.stdout:Created osd(s) 1 on host 'smithi163' 2024-01-24T13:31:58.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:58.022+0000 7fde8bf5c700 1 -- 172.21.15.163:0/2200502853 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fde700a0490 msgr2=0x7fde700a2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:31:58.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:58.022+0000 7fde8bf5c700 1 --2- 172.21.15.163:0/2200502853 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fde700a0490 0x7fde700a2950 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fde84100d30 tx=0x7fde80009260 comp rx=0 tx=0).stop 2024-01-24T13:31:58.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:58.022+0000 7fde8bf5c700 1 -- 172.21.15.163:0/2200502853 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fde84101870 msgr2=0x7fde8419e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:31:58.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:58.022+0000 7fde8bf5c700 1 --2- 172.21.15.163:0/2200502853 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fde84101870 0x7fde8419e060 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fde7c007ee0 tx=0x7fde7c00d3b0 comp rx=0 tx=0).stop 2024-01-24T13:31:58.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:58.022+0000 7fde8bf5c700 1 -- 172.21.15.163:0/2200502853 shutdown_connections 2024-01-24T13:31:58.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:58.022+0000 7fde8bf5c700 1 --2- 172.21.15.163:0/2200502853 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fde84101870 0x7fde8419e060 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:58.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:58.022+0000 7fde8bf5c700 1 --2- 172.21.15.163:0/2200502853 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fde700a0490 0x7fde700a2950 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:58.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:58.022+0000 7fde8bf5c700 1 --2- 172.21.15.163:0/2200502853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde840ffcd0 0x7fde84196aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:58.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:58.022+0000 7fde8bf5c700 1 --2- 172.21.15.163:0/2200502853 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde84100ed0 0x7fde84196fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:31:58.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:58.022+0000 7fde8bf5c700 1 -- 172.21.15.163:0/2200502853 >> 172.21.15.163:0/2200502853 conn(0x7fde840fb250 msgr2=0x7fde84068880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:31:58.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:58.022+0000 7fde8bf5c700 1 -- 172.21.15.163:0/2200502853 shutdown_connections 2024-01-24T13:31:58.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:31:58.022+0000 7fde8bf5c700 1 -- 172.21.15.163:0/2200502853 wait complete. 2024-01-24T13:31:58.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:57 smithi029 ceph-mon[40651]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T13:31:58.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:57 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:58.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:57 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:58.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:57 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:58.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:57 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:31:58.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:57 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:58.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:57 smithi187 ceph-mon[44911]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T13:31:58.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:57 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:58.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:57 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:58.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:57 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:31:58.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:57 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:31:58.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:57 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:58.413 DEBUG:teuthology.orchestra.run.smithi163:osd.1> sudo journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@osd.1.service 2024-01-24T13:31:58.415 INFO:tasks.cephadm:Deploying osd.2 on smithi187 with /dev/vg_nvme/lv_4... 2024-01-24T13:31:58.415 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-01-24T13:31:58.445 INFO:journalctl@ceph.osd.1.smithi163.stdout:-- Logs begin at Wed 2024-01-24 13:14:09 UTC. -- 2024-01-24T13:31:58.640 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.c/config 2024-01-24T13:31:58.915 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:58 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:31:58.915 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:58 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:58.915 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:31:58 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:59.051 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:58 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:31:59.051 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:58 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:59.051 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:31:58 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:59.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:58 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:31:59.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:58 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:59.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:31:58 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:31:59.863 INFO:teuthology.orchestra.run.smithi187.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-01-24T13:31:59.863 INFO:teuthology.orchestra.run.smithi187.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-01-24T13:31:59.863 INFO:teuthology.orchestra.run.smithi187.stderr: stderr: 10+0 records in 2024-01-24T13:31:59.863 INFO:teuthology.orchestra.run.smithi187.stderr:10+0 records out 2024-01-24T13:31:59.863 INFO:teuthology.orchestra.run.smithi187.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112365 s, 933 MB/s 2024-01-24T13:31:59.863 INFO:teuthology.orchestra.run.smithi187.stderr:--> Zapping successful for: 2024-01-24T13:32:00.024 INFO:journalctl@ceph.osd.1.smithi163.stdout:Jan 24 13:31:59 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-1[49943]: 2024-01-24T13:31:59.762+0000 7f77c93bf640 -1 osd.1 0 log_to_monitors true 2024-01-24T13:32:00.245 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:00 smithi187 ceph-mon[44911]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T13:32:00.245 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:00 smithi187 ceph-mon[44911]: from='osd.1 [v2:172.21.15.163:6800/2217244566,v1:172.21.15.163:6801/2217244566]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-01-24T13:32:00.245 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:00 smithi187 ceph-mon[44911]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-01-24T13:32:00.247 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph orch daemon add osd smithi187:vg_nvme/lv_4 2024-01-24T13:32:00.294 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:00 smithi163 ceph-mon[44969]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T13:32:00.294 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:00 smithi163 ceph-mon[44969]: from='osd.1 [v2:172.21.15.163:6800/2217244566,v1:172.21.15.163:6801/2217244566]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-01-24T13:32:00.294 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:00 smithi163 ceph-mon[44969]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-01-24T13:32:00.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:00 smithi029 ceph-mon[40651]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T13:32:00.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:00 smithi029 ceph-mon[40651]: from='osd.1 [v2:172.21.15.163:6800/2217244566,v1:172.21.15.163:6801/2217244566]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-01-24T13:32:00.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:00 smithi029 ceph-mon[40651]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-01-24T13:32:00.462 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.c/config 2024-01-24T13:32:01.185 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:01 smithi187 ceph-mon[44911]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-01-24T13:32:01.185 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:01 smithi187 ceph-mon[44911]: osdmap e11: 2 total, 1 up, 2 in 2024-01-24T13:32:01.185 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:01 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-24T13:32:01.185 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:01 smithi187 ceph-mon[44911]: from='osd.1 [v2:172.21.15.163:6800/2217244566,v1:172.21.15.163:6801/2217244566]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-01-24T13:32:01.185 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:01 smithi187 ceph-mon[44911]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-01-24T13:32:01.185 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:01 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:01.185 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:01 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:01.186 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:01 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:32:01.186 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:01 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:01.186 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:01 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:01.186 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:01 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:32:01.186 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:01 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:01.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:01 smithi029 ceph-mon[40651]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-01-24T13:32:01.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:01 smithi029 ceph-mon[40651]: osdmap e11: 2 total, 1 up, 2 in 2024-01-24T13:32:01.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:01 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-24T13:32:01.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:01 smithi029 ceph-mon[40651]: from='osd.1 [v2:172.21.15.163:6800/2217244566,v1:172.21.15.163:6801/2217244566]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-01-24T13:32:01.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:01 smithi029 ceph-mon[40651]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-01-24T13:32:01.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:01 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:01.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:01 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:01.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:01 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:32:01.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:01 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:01.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:01 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:01.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:01 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:32:01.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:01 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:01.364 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.362+0000 7f709a773700 1 -- 172.21.15.187:0/3615686150 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7094100ed0 msgr2=0x7f7094101330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:32:01.364 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.362+0000 7f709a773700 1 --2- 172.21.15.187:0/3615686150 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7094100ed0 0x7f7094101330 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7088009990 tx=0x7f7088009ca0 comp rx=0 tx=0).stop 2024-01-24T13:32:01.364 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.362+0000 7f709a773700 1 -- 172.21.15.187:0/3615686150 shutdown_connections 2024-01-24T13:32:01.364 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.362+0000 7f709a773700 1 --2- 172.21.15.187:0/3615686150 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7094101870 0x7f7094108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:32:01.364 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.362+0000 7f709a773700 1 --2- 172.21.15.187:0/3615686150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70940ffcd0 0x7f70941000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:32:01.364 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.362+0000 7f709a773700 1 --2- 172.21.15.187:0/3615686150 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7094100ed0 0x7f7094101330 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:32:01.364 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.362+0000 7f709a773700 1 -- 172.21.15.187:0/3615686150 >> 172.21.15.187:0/3615686150 conn(0x7f70940fb250 msgr2=0x7f70940fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:32:01.364 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.362+0000 7f709a773700 1 -- 172.21.15.187:0/3615686150 shutdown_connections 2024-01-24T13:32:01.364 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.362+0000 7f709a773700 1 -- 172.21.15.187:0/3615686150 wait complete. 2024-01-24T13:32:01.364 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.362+0000 7f709a773700 1 Processor -- start 2024-01-24T13:32:01.364 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.362+0000 7f709a773700 1 -- start start 2024-01-24T13:32:01.365 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.363+0000 7f709a773700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70940ffcd0 0x7f7094196af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:32:01.365 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.363+0000 7f709a773700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7094100ed0 0x7f7094197030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:32:01.365 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.363+0000 7f709a773700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7094101870 0x7f709419e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:32:01.365 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.363+0000 7f709a773700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f709419e680 con 0x7f7094101870 2024-01-24T13:32:01.365 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.363+0000 7f709a773700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f709419e7c0 con 0x7f7094100ed0 2024-01-24T13:32:01.365 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.363+0000 7f709a773700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f709419e950 con 0x7f70940ffcd0 2024-01-24T13:32:01.365 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.363+0000 7f7093fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70940ffcd0 0x7f7094196af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:32:01.365 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.363+0000 7f7093fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70940ffcd0 0x7f7094196af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:45694/0 (socket says 172.21.15.187:45694) 2024-01-24T13:32:01.365 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.363+0000 7f7093fff700 1 -- 172.21.15.187:0/4051529375 learned_addr learned my addr 172.21.15.187:0/4051529375 (peer_addr_for_me v2:172.21.15.187:0/0) 2024-01-24T13:32:01.365 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.363+0000 7f70937fe700 1 --2- 172.21.15.187:0/4051529375 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7094100ed0 0x7f7094197030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:32:01.365 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.363+0000 7f7098d10700 1 --2- 172.21.15.187:0/4051529375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7094101870 0x7f709419e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:32:01.366 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.363+0000 7f7093fff700 1 -- 172.21.15.187:0/4051529375 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7094100ed0 msgr2=0x7f7094197030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:32:01.366 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.363+0000 7f7093fff700 1 --2- 172.21.15.187:0/4051529375 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7094100ed0 0x7f7094197030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:32:01.366 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.363+0000 7f7093fff700 1 -- 172.21.15.187:0/4051529375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7094101870 msgr2=0x7f709419e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:32:01.366 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.363+0000 7f7093fff700 1 --2- 172.21.15.187:0/4051529375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7094101870 0x7f709419e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:32:01.366 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.363+0000 7f7093fff700 1 -- 172.21.15.187:0/4051529375 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70880095f0 con 0x7f70940ffcd0 2024-01-24T13:32:01.366 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.363+0000 7f7093fff700 1 --2- 172.21.15.187:0/4051529375 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70940ffcd0 0x7f7094196af0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f708400c9a0 tx=0x7f708400cd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:32:01.366 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.363+0000 7f70917fa700 1 -- 172.21.15.187:0/4051529375 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7084004220 con 0x7f70940ffcd0 2024-01-24T13:32:01.366 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.363+0000 7f70917fa700 1 -- 172.21.15.187:0/4051529375 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7084007d20 con 0x7f70940ffcd0 2024-01-24T13:32:01.366 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.363+0000 7f709a773700 1 -- 172.21.15.187:0/4051529375 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f709419ec30 con 0x7f70940ffcd0 2024-01-24T13:32:01.366 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.363+0000 7f70917fa700 1 -- 172.21.15.187:0/4051529375 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7084003d00 con 0x7f70940ffcd0 2024-01-24T13:32:01.366 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.364+0000 7f709a773700 1 -- 172.21.15.187:0/4051529375 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f709419f180 con 0x7f70940ffcd0 2024-01-24T13:32:01.367 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.365+0000 7f709a773700 1 -- 172.21.15.187:0/4051529375 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7094107db0 con 0x7f70940ffcd0 2024-01-24T13:32:01.367 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.365+0000 7f70917fa700 1 -- 172.21.15.187:0/4051529375 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 15) v1 ==== 135014+0+0 (secure 0 0 0) 0x7f7084004380 con 0x7f70940ffcd0 2024-01-24T13:32:01.367 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.366+0000 7f70917fa700 1 --2- 172.21.15.187:0/4051529375 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f707c0a0510 0x7f707c0a29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:32:01.367 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.366+0000 7f70917fa700 1 -- 172.21.15.187:0/4051529375 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2029+0+0 (secure 0 0 0) 0x7f70840c9bd0 con 0x7f70940ffcd0 2024-01-24T13:32:01.368 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.366+0000 7f70937fe700 1 --2- 172.21.15.187:0/4051529375 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f707c0a0510 0x7f707c0a29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:32:01.368 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.367+0000 7f70937fe700 1 --2- 172.21.15.187:0/4051529375 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f707c0a0510 0x7f707c0a29d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f708800b5c0 tx=0x7f7088005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:32:01.370 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.370+0000 7f70917fa700 1 -- 172.21.15.187:0/4051529375 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f70840950e0 con 0x7f70940ffcd0 2024-01-24T13:32:01.482 INFO:journalctl@ceph.osd.1.smithi163.stdout:Jan 24 13:32:01 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-1[49943]: 2024-01-24T13:32:01.031+0000 7f77bea24700 -1 osd.1 0 waiting for initial osdmap 2024-01-24T13:32:01.482 INFO:journalctl@ceph.osd.1.smithi163.stdout:Jan 24 13:32:01 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-1[49943]: 2024-01-24T13:32:01.034+0000 7f77b8815700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-24T13:32:01.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:01 smithi163 ceph-mon[44969]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-01-24T13:32:01.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:01 smithi163 ceph-mon[44969]: osdmap e11: 2 total, 1 up, 2 in 2024-01-24T13:32:01.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:01 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-24T13:32:01.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:01 smithi163 ceph-mon[44969]: from='osd.1 [v2:172.21.15.163:6800/2217244566,v1:172.21.15.163:6801/2217244566]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-01-24T13:32:01.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:01 smithi163 ceph-mon[44969]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-01-24T13:32:01.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:01 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:01.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:01 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:01.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:01 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:32:01.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:01 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:01.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:01 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:01.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:01 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:32:01.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:01 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:01.565 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:01.566+0000 7f709a773700 1 -- 172.21.15.187:0/4051529375 --> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f7094061180 con 0x7f707c0a0510 2024-01-24T13:32:02.225 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:02 smithi187 ceph-mon[44911]: Detected new or changed devices on smithi163 2024-01-24T13:32:02.225 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:02 smithi187 ceph-mon[44911]: Adjusting osd_memory_target on smithi163 to 17114M 2024-01-24T13:32:02.225 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:02 smithi187 ceph-mon[44911]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T13:32:02.225 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:02 smithi187 ceph-mon[44911]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-01-24T13:32:02.226 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:02 smithi187 ceph-mon[44911]: osdmap e12: 2 total, 1 up, 2 in 2024-01-24T13:32:02.226 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:02 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-24T13:32:02.226 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:02 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-24T13:32:02.226 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:02 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T13:32:02.226 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:02 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T13:32:02.226 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:02 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:02.351 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:02 smithi029 ceph-mon[40651]: Detected new or changed devices on smithi163 2024-01-24T13:32:02.351 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:02 smithi029 ceph-mon[40651]: Adjusting osd_memory_target on smithi163 to 17114M 2024-01-24T13:32:02.352 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:02 smithi029 ceph-mon[40651]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T13:32:02.352 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:02 smithi029 ceph-mon[40651]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-01-24T13:32:02.352 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:02 smithi029 ceph-mon[40651]: osdmap e12: 2 total, 1 up, 2 in 2024-01-24T13:32:02.352 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:02 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-24T13:32:02.352 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:02 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-24T13:32:02.352 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:02 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T13:32:02.352 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:02 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T13:32:02.352 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:02 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:02.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:02 smithi163 ceph-mon[44969]: Detected new or changed devices on smithi163 2024-01-24T13:32:02.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:02 smithi163 ceph-mon[44969]: Adjusting osd_memory_target on smithi163 to 17114M 2024-01-24T13:32:02.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:02 smithi163 ceph-mon[44969]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T13:32:02.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:02 smithi163 ceph-mon[44969]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-01-24T13:32:02.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:02 smithi163 ceph-mon[44969]: osdmap e12: 2 total, 1 up, 2 in 2024-01-24T13:32:02.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:02 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-24T13:32:02.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:02 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-24T13:32:02.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:02 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T13:32:02.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:02 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T13:32:02.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:02 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:03.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:03 smithi187 ceph-mon[44911]: purged_snaps scrub starts 2024-01-24T13:32:03.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:03 smithi187 ceph-mon[44911]: purged_snaps scrub ok 2024-01-24T13:32:03.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:03 smithi187 ceph-mon[44911]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:32:03.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:03 smithi187 ceph-mon[44911]: osd.1 [v2:172.21.15.163:6800/2217244566,v1:172.21.15.163:6801/2217244566] boot 2024-01-24T13:32:03.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:03 smithi187 ceph-mon[44911]: osdmap e13: 2 total, 2 up, 2 in 2024-01-24T13:32:03.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:03 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-24T13:32:03.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:03 smithi029 ceph-mon[40651]: purged_snaps scrub starts 2024-01-24T13:32:03.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:03 smithi029 ceph-mon[40651]: purged_snaps scrub ok 2024-01-24T13:32:03.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:03 smithi029 ceph-mon[40651]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:32:03.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:03 smithi029 ceph-mon[40651]: osd.1 [v2:172.21.15.163:6800/2217244566,v1:172.21.15.163:6801/2217244566] boot 2024-01-24T13:32:03.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:03 smithi029 ceph-mon[40651]: osdmap e13: 2 total, 2 up, 2 in 2024-01-24T13:32:03.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:03 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-24T13:32:03.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:03 smithi163 ceph-mon[44969]: purged_snaps scrub starts 2024-01-24T13:32:03.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:03 smithi163 ceph-mon[44969]: purged_snaps scrub ok 2024-01-24T13:32:03.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:03 smithi163 ceph-mon[44969]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:32:03.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:03 smithi163 ceph-mon[44969]: osd.1 [v2:172.21.15.163:6800/2217244566,v1:172.21.15.163:6801/2217244566] boot 2024-01-24T13:32:03.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:03 smithi163 ceph-mon[44969]: osdmap e13: 2 total, 2 up, 2 in 2024-01-24T13:32:03.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:03 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-24T13:32:04.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:04 smithi187 ceph-mon[44911]: pgmap v49: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T13:32:04.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:04 smithi187 ceph-mon[44911]: osdmap e14: 2 total, 2 up, 2 in 2024-01-24T13:32:04.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:04 smithi187 ceph-mon[44911]: from='client.? 172.21.15.187:0/3079950887' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f0ed4131-cd29-4706-89ea-fde40f46ea83"}]: dispatch 2024-01-24T13:32:04.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:04 smithi187 ceph-mon[44911]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f0ed4131-cd29-4706-89ea-fde40f46ea83"}]: dispatch 2024-01-24T13:32:04.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:04 smithi187 ceph-mon[44911]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f0ed4131-cd29-4706-89ea-fde40f46ea83"}]': finished 2024-01-24T13:32:04.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:04 smithi187 ceph-mon[44911]: osdmap e15: 3 total, 2 up, 3 in 2024-01-24T13:32:04.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:04 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-24T13:32:04.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:04 smithi187 ceph-mon[44911]: from='client.? 172.21.15.187:0/3620526240' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T13:32:04.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:04 smithi029 ceph-mon[40651]: pgmap v49: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T13:32:04.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:04 smithi029 ceph-mon[40651]: osdmap e14: 2 total, 2 up, 2 in 2024-01-24T13:32:04.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:04 smithi029 ceph-mon[40651]: from='client.? 172.21.15.187:0/3079950887' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f0ed4131-cd29-4706-89ea-fde40f46ea83"}]: dispatch 2024-01-24T13:32:04.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:04 smithi029 ceph-mon[40651]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f0ed4131-cd29-4706-89ea-fde40f46ea83"}]: dispatch 2024-01-24T13:32:04.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:04 smithi029 ceph-mon[40651]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f0ed4131-cd29-4706-89ea-fde40f46ea83"}]': finished 2024-01-24T13:32:04.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:04 smithi029 ceph-mon[40651]: osdmap e15: 3 total, 2 up, 3 in 2024-01-24T13:32:04.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:04 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-24T13:32:04.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:04 smithi029 ceph-mon[40651]: from='client.? 172.21.15.187:0/3620526240' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T13:32:04.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:04 smithi163 ceph-mon[44969]: pgmap v49: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T13:32:04.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:04 smithi163 ceph-mon[44969]: osdmap e14: 2 total, 2 up, 2 in 2024-01-24T13:32:04.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:04 smithi163 ceph-mon[44969]: from='client.? 172.21.15.187:0/3079950887' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f0ed4131-cd29-4706-89ea-fde40f46ea83"}]: dispatch 2024-01-24T13:32:04.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:04 smithi163 ceph-mon[44969]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f0ed4131-cd29-4706-89ea-fde40f46ea83"}]: dispatch 2024-01-24T13:32:04.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:04 smithi163 ceph-mon[44969]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f0ed4131-cd29-4706-89ea-fde40f46ea83"}]': finished 2024-01-24T13:32:04.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:04 smithi163 ceph-mon[44969]: osdmap e15: 3 total, 2 up, 3 in 2024-01-24T13:32:04.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:04 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-24T13:32:04.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:04 smithi163 ceph-mon[44969]: from='client.? 172.21.15.187:0/3620526240' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T13:32:06.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:06 smithi187 ceph-mon[44911]: pgmap v52: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T13:32:06.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:06 smithi029 ceph-mon[40651]: pgmap v52: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T13:32:06.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:06 smithi163 ceph-mon[44969]: pgmap v52: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T13:32:08.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:08 smithi187 ceph-mon[44911]: pgmap v53: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T13:32:08.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:08 smithi029 ceph-mon[40651]: pgmap v53: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T13:32:08.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:08 smithi163 ceph-mon[44969]: pgmap v53: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T13:32:10.233 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:10 smithi187 ceph-mon[44911]: pgmap v54: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T13:32:10.233 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:10 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-01-24T13:32:10.233 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:10 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:10.233 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:10 smithi187 ceph-mon[44911]: Deploying daemon osd.2 on smithi187 2024-01-24T13:32:10.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:10 smithi029 ceph-mon[40651]: pgmap v54: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T13:32:10.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:10 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-01-24T13:32:10.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:10 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:10.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:10 smithi029 ceph-mon[40651]: Deploying daemon osd.2 on smithi187 2024-01-24T13:32:10.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:10 smithi163 ceph-mon[44969]: pgmap v54: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T13:32:10.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:10 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-01-24T13:32:10.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:10 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:10.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:10 smithi163 ceph-mon[44969]: Deploying daemon osd.2 on smithi187 2024-01-24T13:32:12.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:12 smithi187 ceph-mon[44911]: pgmap v55: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T13:32:12.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:12 smithi029 ceph-mon[40651]: pgmap v55: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T13:32:12.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:12 smithi163 ceph-mon[44969]: pgmap v55: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T13:32:14.015 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:13 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:14.015 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:13 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:32:14.015 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:13 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:14.015 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:13 smithi187 ceph-mon[44911]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T13:32:14.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:13 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:14.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:13 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:32:14.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:13 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:14.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:13 smithi029 ceph-mon[40651]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T13:32:14.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:13 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:14.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:13 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:32:14.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:13 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:14.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:13 smithi163 ceph-mon[44969]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T13:32:14.872 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:14.870+0000 7f70917fa700 1 -- 172.21.15.187:0/4051529375 <== mgr.14152 v2:172.21.15.29:6800/2498991338 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7094061180 con 0x7f707c0a0510 2024-01-24T13:32:14.872 INFO:teuthology.orchestra.run.smithi187.stdout:Created osd(s) 2 on host 'smithi187' 2024-01-24T13:32:14.874 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:14.873+0000 7f709a773700 1 -- 172.21.15.187:0/4051529375 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f707c0a0510 msgr2=0x7f707c0a29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:32:14.874 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:14.873+0000 7f709a773700 1 --2- 172.21.15.187:0/4051529375 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f707c0a0510 0x7f707c0a29d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f708800b5c0 tx=0x7f7088005fb0 comp rx=0 tx=0).stop 2024-01-24T13:32:14.874 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:14.873+0000 7f709a773700 1 -- 172.21.15.187:0/4051529375 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70940ffcd0 msgr2=0x7f7094196af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:32:14.874 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:14.873+0000 7f709a773700 1 --2- 172.21.15.187:0/4051529375 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70940ffcd0 0x7f7094196af0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f708400c9a0 tx=0x7f708400cd60 comp rx=0 tx=0).stop 2024-01-24T13:32:14.874 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:14.873+0000 7f709a773700 1 -- 172.21.15.187:0/4051529375 shutdown_connections 2024-01-24T13:32:14.874 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:14.873+0000 7f709a773700 1 --2- 172.21.15.187:0/4051529375 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f707c0a0510 0x7f707c0a29d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:32:14.874 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:14.873+0000 7f709a773700 1 --2- 172.21.15.187:0/4051529375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7094101870 0x7f709419e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:32:14.874 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:14.873+0000 7f709a773700 1 --2- 172.21.15.187:0/4051529375 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70940ffcd0 0x7f7094196af0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:32:14.874 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:14.873+0000 7f709a773700 1 --2- 172.21.15.187:0/4051529375 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7094100ed0 0x7f7094197030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:32:14.874 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:14.873+0000 7f709a773700 1 -- 172.21.15.187:0/4051529375 >> 172.21.15.187:0/4051529375 conn(0x7f70940fb250 msgr2=0x7f70940fd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:32:14.874 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:14.873+0000 7f709a773700 1 -- 172.21.15.187:0/4051529375 shutdown_connections 2024-01-24T13:32:14.875 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:14.873+0000 7f709a773700 1 -- 172.21.15.187:0/4051529375 wait complete. 2024-01-24T13:32:15.198 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:14 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:15.199 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:14 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:15.199 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:14 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:15.199 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:14 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:32:15.199 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:14 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:15.199 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:14 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:15.199 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:14 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:15.199 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:14 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:32:15.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:14 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:15.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:14 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:15.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:14 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:15.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:14 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:32:15.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:14 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:15.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:14 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:15.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:14 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:15.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:14 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:32:15.291 DEBUG:teuthology.orchestra.run.smithi187:osd.2> sudo journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@osd.2.service 2024-01-24T13:32:15.293 INFO:tasks.cephadm:Deploying osd.3 on smithi029 with /dev/vg_nvme/lv_3... 2024-01-24T13:32:15.293 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-01-24T13:32:15.323 INFO:journalctl@ceph.osd.2.smithi187.stdout:-- Logs begin at Wed 2024-01-24 13:13:23 UTC. -- 2024-01-24T13:32:15.328 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:14 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:15.328 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:14 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:15.328 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:14 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:15.328 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:14 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:32:15.328 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:14 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:15.328 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:14 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:15.328 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:14 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:15.328 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:14 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:32:15.549 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T13:32:16.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:16 smithi187 ceph-mon[44911]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T13:32:16.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:16 smithi029 ceph-mon[40651]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T13:32:16.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:16 smithi163 ceph-mon[44969]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T13:32:16.926 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-01-24T13:32:16.927 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-01-24T13:32:16.927 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10+0 records in 2024-01-24T13:32:16.927 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2024-01-24T13:32:16.927 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0162306 s, 646 MB/s 2024-01-24T13:32:16.927 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping successful for: 2024-01-24T13:32:17.030 INFO:journalctl@ceph.osd.2.smithi187.stdout:Jan 24 13:32:16 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-2[50708]: 2024-01-24T13:32:16.701+0000 7f07d4ea5640 -1 osd.2 0 log_to_monitors true 2024-01-24T13:32:17.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:17 smithi187 ceph-mon[44911]: from='osd.2 [v2:172.21.15.187:6800/3589694992,v1:172.21.15.187:6801/3589694992]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-01-24T13:32:17.326 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:17 smithi029 ceph-mon[40651]: from='osd.2 [v2:172.21.15.187:6800/3589694992,v1:172.21.15.187:6801/3589694992]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-01-24T13:32:17.328 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph orch daemon add osd smithi029:vg_nvme/lv_3 2024-01-24T13:32:17.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:17 smithi163 ceph-mon[44969]: from='osd.2 [v2:172.21.15.187:6800/3589694992,v1:172.21.15.187:6801/3589694992]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-01-24T13:32:17.624 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T13:32:18.304 INFO:journalctl@ceph.osd.2.smithi187.stdout:Jan 24 13:32:18 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-2[50708]: 2024-01-24T13:32:18.042+0000 7f07ca50a700 -1 osd.2 0 waiting for initial osdmap 2024-01-24T13:32:18.305 INFO:journalctl@ceph.osd.2.smithi187.stdout:Jan 24 13:32:18 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-2[50708]: 2024-01-24T13:32:18.049+0000 7f07c4afc700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-24T13:32:18.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:18 smithi187 ceph-mon[44911]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T13:32:18.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:18 smithi187 ceph-mon[44911]: from='osd.2 [v2:172.21.15.187:6800/3589694992,v1:172.21.15.187:6801/3589694992]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-01-24T13:32:18.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:18 smithi187 ceph-mon[44911]: osdmap e16: 3 total, 2 up, 3 in 2024-01-24T13:32:18.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:18 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-24T13:32:18.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:18 smithi187 ceph-mon[44911]: from='osd.2 [v2:172.21.15.187:6800/3589694992,v1:172.21.15.187:6801/3589694992]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-01-24T13:32:18.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:18 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:18.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:18 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:18.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:18 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:32:18.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:18 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:18.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:18 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:18.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:18 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:32:18.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:18 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:18.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:18 smithi029 ceph-mon[40651]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T13:32:18.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:18 smithi029 ceph-mon[40651]: from='osd.2 [v2:172.21.15.187:6800/3589694992,v1:172.21.15.187:6801/3589694992]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-01-24T13:32:18.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:18 smithi029 ceph-mon[40651]: osdmap e16: 3 total, 2 up, 3 in 2024-01-24T13:32:18.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:18 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-24T13:32:18.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:18 smithi029 ceph-mon[40651]: from='osd.2 [v2:172.21.15.187:6800/3589694992,v1:172.21.15.187:6801/3589694992]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-01-24T13:32:18.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:18 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:18.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:18 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:18.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:18 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:32:18.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:18 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:18.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:18 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:18.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:18 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:32:18.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:18 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:18.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:18 smithi163 ceph-mon[44969]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T13:32:18.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:18 smithi163 ceph-mon[44969]: from='osd.2 [v2:172.21.15.187:6800/3589694992,v1:172.21.15.187:6801/3589694992]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-01-24T13:32:18.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:18 smithi163 ceph-mon[44969]: osdmap e16: 3 total, 2 up, 3 in 2024-01-24T13:32:18.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:18 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-24T13:32:18.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:18 smithi163 ceph-mon[44969]: from='osd.2 [v2:172.21.15.187:6800/3589694992,v1:172.21.15.187:6801/3589694992]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-01-24T13:32:18.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:18 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:18.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:18 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:18.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:18 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:32:18.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:18 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:18.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:18 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:18.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:18 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:32:18.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:18 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:18.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.628+0000 7f9bfcff4700 1 -- 172.21.15.29:0/580260474 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf8100ed0 msgr2=0x7f9bf8101330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:32:18.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.628+0000 7f9bfcff4700 1 --2- 172.21.15.29:0/580260474 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf8100ed0 0x7f9bf8101330 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9bec009920 tx=0x7f9bec009c30 comp rx=0 tx=0).stop 2024-01-24T13:32:18.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.628+0000 7f9bfcff4700 1 -- 172.21.15.29:0/580260474 shutdown_connections 2024-01-24T13:32:18.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.628+0000 7f9bfcff4700 1 --2- 172.21.15.29:0/580260474 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9bf8101870 0x7f9bf8108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:32:18.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.628+0000 7f9bfcff4700 1 --2- 172.21.15.29:0/580260474 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9bf80ffcd0 0x7f9bf81000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:32:18.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.628+0000 7f9bfcff4700 1 --2- 172.21.15.29:0/580260474 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf8100ed0 0x7f9bf8101330 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:32:18.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.628+0000 7f9bfcff4700 1 -- 172.21.15.29:0/580260474 >> 172.21.15.29:0/580260474 conn(0x7f9bf80fb250 msgr2=0x7f9bf80fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:32:18.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.628+0000 7f9bfcff4700 1 -- 172.21.15.29:0/580260474 shutdown_connections 2024-01-24T13:32:18.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.628+0000 7f9bfcff4700 1 -- 172.21.15.29:0/580260474 wait complete. 2024-01-24T13:32:18.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.629+0000 7f9bfcff4700 1 Processor -- start 2024-01-24T13:32:18.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.629+0000 7f9bfcff4700 1 -- start start 2024-01-24T13:32:18.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.629+0000 7f9bfcff4700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9bf80ffcd0 0x7f9bf8104070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:32:18.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.629+0000 7f9bfcff4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9bf8100ed0 0x7f9bf81026c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:32:18.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.629+0000 7f9bfcff4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf8101870 0x7f9bf8102c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:32:18.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.629+0000 7f9bfcff4700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf8103380 con 0x7f9bf8101870 2024-01-24T13:32:18.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.629+0000 7f9bfcff4700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf81034f0 con 0x7f9bf80ffcd0 2024-01-24T13:32:18.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.629+0000 7f9bfcff4700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf81a0ba0 con 0x7f9bf8100ed0 2024-01-24T13:32:18.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.630+0000 7f9bf6d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf8101870 0x7f9bf8102c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:32:18.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.630+0000 7f9bf6d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf8101870 0x7f9bf8102c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44640/0 (socket says 172.21.15.29:44640) 2024-01-24T13:32:18.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.630+0000 7f9bf6d9d700 1 -- 172.21.15.29:0/1413489584 learned_addr learned my addr 172.21.15.29:0/1413489584 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:32:18.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.630+0000 7f9bf5d9b700 1 --2- 172.21.15.29:0/1413489584 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9bf8100ed0 0x7f9bf81026c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:32:18.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.630+0000 7f9bf659c700 1 --2- 172.21.15.29:0/1413489584 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9bf80ffcd0 0x7f9bf8104070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:32:18.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.630+0000 7f9bf6d9d700 1 -- 172.21.15.29:0/1413489584 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9bf8100ed0 msgr2=0x7f9bf81026c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:32:18.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.630+0000 7f9bf6d9d700 1 --2- 172.21.15.29:0/1413489584 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9bf8100ed0 0x7f9bf81026c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:32:18.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.630+0000 7f9bf6d9d700 1 -- 172.21.15.29:0/1413489584 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9bf80ffcd0 msgr2=0x7f9bf8104070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:32:18.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.630+0000 7f9bf6d9d700 1 --2- 172.21.15.29:0/1413489584 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9bf80ffcd0 0x7f9bf8104070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:32:18.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.630+0000 7f9bf6d9d700 1 -- 172.21.15.29:0/1413489584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bec009520 con 0x7f9bf8101870 2024-01-24T13:32:18.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.630+0000 7f9bf659c700 1 --2- 172.21.15.29:0/1413489584 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9bf80ffcd0 0x7f9bf8104070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T13:32:18.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.630+0000 7f9bf6d9d700 1 --2- 172.21.15.29:0/1413489584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf8101870 0x7f9bf8102c00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9bf000c110 tx=0x7f9bf000c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:32:18.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.630+0000 7f9be77fe700 1 -- 172.21.15.29:0/1413489584 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf0017070 con 0x7f9bf8101870 2024-01-24T13:32:18.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.630+0000 7f9bfcff4700 1 -- 172.21.15.29:0/1413489584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bf81a0e20 con 0x7f9bf8101870 2024-01-24T13:32:18.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.630+0000 7f9be77fe700 1 -- 172.21.15.29:0/1413489584 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9bf000e050 con 0x7f9bf8101870 2024-01-24T13:32:18.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.630+0000 7f9be77fe700 1 -- 172.21.15.29:0/1413489584 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf0007940 con 0x7f9bf8101870 2024-01-24T13:32:18.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.630+0000 7f9bfcff4700 1 -- 172.21.15.29:0/1413489584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bf81a1240 con 0x7f9bf8101870 2024-01-24T13:32:18.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.632+0000 7f9bfcff4700 1 -- 172.21.15.29:0/1413489584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bf8107ec0 con 0x7f9bf8101870 2024-01-24T13:32:18.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.633+0000 7f9be77fe700 1 -- 172.21.15.29:0/1413489584 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 15) v1 ==== 135014+0+0 (secure 0 0 0) 0x7f9bf0004ad0 con 0x7f9bf8101870 2024-01-24T13:32:18.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.633+0000 7f9be77fe700 1 --2- 172.21.15.29:0/1413489584 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f9be00a0490 0x7f9be00a2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:32:18.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.634+0000 7f9bf659c700 1 --2- 172.21.15.29:0/1413489584 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f9be00a0490 0x7f9be00a2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:32:18.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.634+0000 7f9be77fe700 1 -- 172.21.15.29:0/1413489584 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 2571+0+0 (secure 0 0 0) 0x7f9bf00c9cd0 con 0x7f9bf8101870 2024-01-24T13:32:18.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.634+0000 7f9bf659c700 1 --2- 172.21.15.29:0/1413489584 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f9be00a0490 0x7f9be00a2950 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9bf8100d30 tx=0x7f9be8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:32:18.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.638+0000 7f9be77fe700 1 -- 172.21.15.29:0/1413489584 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f9bf0094fc0 con 0x7f9bf8101870 2024-01-24T13:32:18.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:18.858+0000 7f9bfcff4700 1 -- 172.21.15.29:0/1413489584 --> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f9bf8061180 con 0x7f9be00a0490 2024-01-24T13:32:19.307 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:19 smithi187 ceph-mon[44911]: Detected new or changed devices on smithi187 2024-01-24T13:32:19.307 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:19 smithi187 ceph-mon[44911]: Adjusting osd_memory_target on smithi187 to 17113M 2024-01-24T13:32:19.307 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:19 smithi187 ceph-mon[44911]: from='osd.2 [v2:172.21.15.187:6800/3589694992,v1:172.21.15.187:6801/3589694992]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-01-24T13:32:19.307 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:19 smithi187 ceph-mon[44911]: osdmap e17: 3 total, 2 up, 3 in 2024-01-24T13:32:19.308 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:19 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-24T13:32:19.308 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:19 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-24T13:32:19.308 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:19 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T13:32:19.308 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:19 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T13:32:19.308 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:19 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:19.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:19 smithi029 ceph-mon[40651]: Detected new or changed devices on smithi187 2024-01-24T13:32:19.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:19 smithi029 ceph-mon[40651]: Adjusting osd_memory_target on smithi187 to 17113M 2024-01-24T13:32:19.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:19 smithi029 ceph-mon[40651]: from='osd.2 [v2:172.21.15.187:6800/3589694992,v1:172.21.15.187:6801/3589694992]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-01-24T13:32:19.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:19 smithi029 ceph-mon[40651]: osdmap e17: 3 total, 2 up, 3 in 2024-01-24T13:32:19.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:19 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-24T13:32:19.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:19 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-24T13:32:19.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:19 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T13:32:19.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:19 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T13:32:19.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:19 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:19.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:19 smithi163 ceph-mon[44969]: Detected new or changed devices on smithi187 2024-01-24T13:32:19.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:19 smithi163 ceph-mon[44969]: Adjusting osd_memory_target on smithi187 to 17113M 2024-01-24T13:32:19.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:19 smithi163 ceph-mon[44969]: from='osd.2 [v2:172.21.15.187:6800/3589694992,v1:172.21.15.187:6801/3589694992]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-01-24T13:32:19.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:19 smithi163 ceph-mon[44969]: osdmap e17: 3 total, 2 up, 3 in 2024-01-24T13:32:19.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:19 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-24T13:32:19.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:19 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-24T13:32:19.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:19 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T13:32:19.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:19 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T13:32:19.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:19 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:20.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:20 smithi029 ceph-mon[40651]: purged_snaps scrub starts 2024-01-24T13:32:20.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:20 smithi029 ceph-mon[40651]: purged_snaps scrub ok 2024-01-24T13:32:20.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:20 smithi029 ceph-mon[40651]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T13:32:20.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:20 smithi029 ceph-mon[40651]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:32:20.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:20 smithi029 ceph-mon[40651]: osd.2 [v2:172.21.15.187:6800/3589694992,v1:172.21.15.187:6801/3589694992] boot 2024-01-24T13:32:20.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:20 smithi029 ceph-mon[40651]: osdmap e18: 3 total, 3 up, 3 in 2024-01-24T13:32:20.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:20 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-24T13:32:20.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:20 smithi163 ceph-mon[44969]: purged_snaps scrub starts 2024-01-24T13:32:20.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:20 smithi163 ceph-mon[44969]: purged_snaps scrub ok 2024-01-24T13:32:20.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:20 smithi163 ceph-mon[44969]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T13:32:20.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:20 smithi163 ceph-mon[44969]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:32:20.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:20 smithi163 ceph-mon[44969]: osd.2 [v2:172.21.15.187:6800/3589694992,v1:172.21.15.187:6801/3589694992] boot 2024-01-24T13:32:20.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:20 smithi163 ceph-mon[44969]: osdmap e18: 3 total, 3 up, 3 in 2024-01-24T13:32:20.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:20 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-24T13:32:20.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:20 smithi187 ceph-mon[44911]: purged_snaps scrub starts 2024-01-24T13:32:20.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:20 smithi187 ceph-mon[44911]: purged_snaps scrub ok 2024-01-24T13:32:20.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:20 smithi187 ceph-mon[44911]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T13:32:20.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:20 smithi187 ceph-mon[44911]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:32:20.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:20 smithi187 ceph-mon[44911]: osd.2 [v2:172.21.15.187:6800/3589694992,v1:172.21.15.187:6801/3589694992] boot 2024-01-24T13:32:20.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:20 smithi187 ceph-mon[44911]: osdmap e18: 3 total, 3 up, 3 in 2024-01-24T13:32:20.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:20 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-24T13:32:21.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:21 smithi187 ceph-mon[44911]: osdmap e19: 3 total, 3 up, 3 in 2024-01-24T13:32:21.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:21 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/1698528874' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da821310-b502-4b77-a5ce-0ffed0a3dafc"}]: dispatch 2024-01-24T13:32:21.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:21 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/1698528874' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "da821310-b502-4b77-a5ce-0ffed0a3dafc"}]': finished 2024-01-24T13:32:21.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:21 smithi187 ceph-mon[44911]: osdmap e20: 4 total, 3 up, 4 in 2024-01-24T13:32:21.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:21 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T13:32:21.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:21 smithi029 ceph-mon[40651]: osdmap e19: 3 total, 3 up, 3 in 2024-01-24T13:32:21.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:21 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/1698528874' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da821310-b502-4b77-a5ce-0ffed0a3dafc"}]: dispatch 2024-01-24T13:32:21.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:21 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/1698528874' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "da821310-b502-4b77-a5ce-0ffed0a3dafc"}]': finished 2024-01-24T13:32:21.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:21 smithi029 ceph-mon[40651]: osdmap e20: 4 total, 3 up, 4 in 2024-01-24T13:32:21.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:21 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T13:32:21.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:21 smithi163 ceph-mon[44969]: osdmap e19: 3 total, 3 up, 3 in 2024-01-24T13:32:21.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:21 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/1698528874' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da821310-b502-4b77-a5ce-0ffed0a3dafc"}]: dispatch 2024-01-24T13:32:21.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:21 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/1698528874' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "da821310-b502-4b77-a5ce-0ffed0a3dafc"}]': finished 2024-01-24T13:32:21.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:21 smithi163 ceph-mon[44969]: osdmap e20: 4 total, 3 up, 4 in 2024-01-24T13:32:21.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:21 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T13:32:22.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:22 smithi187 ceph-mon[44911]: pgmap v65: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-01-24T13:32:22.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:22 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-01-24T13:32:22.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:22 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/2337128437' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T13:32:22.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:22 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-01-24T13:32:22.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:22 smithi187 ceph-mon[44911]: osdmap e21: 4 total, 3 up, 4 in 2024-01-24T13:32:22.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:22 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T13:32:22.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:22 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-01-24T13:32:22.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:22 smithi029 ceph-mon[40651]: pgmap v65: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-01-24T13:32:22.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:22 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-01-24T13:32:22.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:22 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/2337128437' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T13:32:22.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:22 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-01-24T13:32:22.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:22 smithi029 ceph-mon[40651]: osdmap e21: 4 total, 3 up, 4 in 2024-01-24T13:32:22.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:22 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T13:32:22.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:22 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-01-24T13:32:22.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:22 smithi163 ceph-mon[44969]: pgmap v65: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-01-24T13:32:22.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:22 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-01-24T13:32:22.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:22 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/2337128437' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T13:32:22.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:22 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-01-24T13:32:22.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:22 smithi163 ceph-mon[44969]: osdmap e21: 4 total, 3 up, 4 in 2024-01-24T13:32:22.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:22 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T13:32:22.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:22 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-01-24T13:32:23.599 INFO:journalctl@ceph.osd.0.smithi029.stdout:Jan 24 13:32:23 smithi029 sudo[66478]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-01-24T13:32:23.599 INFO:journalctl@ceph.osd.0.smithi029.stdout:Jan 24 13:32:23 smithi029 sudo[66478]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-24T13:32:23.599 INFO:journalctl@ceph.osd.0.smithi029.stdout:Jan 24 13:32:23 smithi029 sudo[66478]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-24T13:32:23.599 INFO:journalctl@ceph.osd.0.smithi029.stdout:Jan 24 13:32:23 smithi029 sudo[66478]: pam_unix(sudo:session): session closed for user root 2024-01-24T13:32:23.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:23.742+0000 7f9be77fe700 1 -- 172.21.15.29:0/1413489584 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f9bf0095390 con 0x7f9bf8101870 2024-01-24T13:32:24.035 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:23 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-01-24T13:32:24.035 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:23 smithi163 ceph-mon[44969]: osdmap e22: 4 total, 3 up, 4 in 2024-01-24T13:32:24.035 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:23 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T13:32:24.035 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:23 smithi163 ceph-mon[44969]: pgmap v68: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-01-24T13:32:24.054 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:23 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-01-24T13:32:24.054 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:23 smithi187 ceph-mon[44911]: osdmap e22: 4 total, 3 up, 4 in 2024-01-24T13:32:24.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:23 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T13:32:24.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:23 smithi187 ceph-mon[44911]: pgmap v68: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-01-24T13:32:24.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:23 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-01-24T13:32:24.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:23 smithi029 ceph-mon[40651]: osdmap e22: 4 total, 3 up, 4 in 2024-01-24T13:32:24.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:23 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T13:32:24.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:23 smithi029 ceph-mon[40651]: pgmap v68: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-01-24T13:32:24.099 INFO:journalctl@ceph.osd.0.smithi029.stdout:Jan 24 13:32:23 smithi029 sudo[66667]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-01-24T13:32:24.099 INFO:journalctl@ceph.osd.0.smithi029.stdout:Jan 24 13:32:23 smithi029 sudo[66667]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-24T13:32:24.099 INFO:journalctl@ceph.osd.0.smithi029.stdout:Jan 24 13:32:23 smithi029 sudo[66667]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-24T13:32:24.099 INFO:journalctl@ceph.osd.0.smithi029.stdout:Jan 24 13:32:23 smithi029 sudo[66667]: pam_unix(sudo:session): session closed for user root 2024-01-24T13:32:24.482 INFO:journalctl@ceph.osd.1.smithi163.stdout:Jan 24 13:32:24 smithi163 sudo[55528]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-01-24T13:32:24.482 INFO:journalctl@ceph.osd.1.smithi163.stdout:Jan 24 13:32:24 smithi163 sudo[55528]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-24T13:32:24.482 INFO:journalctl@ceph.osd.1.smithi163.stdout:Jan 24 13:32:24 smithi163 sudo[55528]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-24T13:32:24.482 INFO:journalctl@ceph.osd.1.smithi163.stdout:Jan 24 13:32:24 smithi163 sudo[55528]: pam_unix(sudo:session): session closed for user root 2024-01-24T13:32:24.482 INFO:journalctl@ceph.osd.1.smithi163.stdout:Jan 24 13:32:24 smithi163 sudo[55531]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-01-24T13:32:24.482 INFO:journalctl@ceph.osd.1.smithi163.stdout:Jan 24 13:32:24 smithi163 sudo[55531]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-24T13:32:24.483 INFO:journalctl@ceph.osd.1.smithi163.stdout:Jan 24 13:32:24 smithi163 sudo[55531]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-24T13:32:24.483 INFO:journalctl@ceph.osd.1.smithi163.stdout:Jan 24 13:32:24 smithi163 sudo[55531]: pam_unix(sudo:session): session closed for user root 2024-01-24T13:32:24.804 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:24 smithi187 ceph-mon[44911]: osdmap e23: 4 total, 3 up, 4 in 2024-01-24T13:32:24.804 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:24 smithi187 ceph-mon[44911]: mgrmap e16: x(active, since 113s), standbys: z, y 2024-01-24T13:32:24.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:24 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T13:32:24.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:24 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:24.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:24 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:24.805 INFO:journalctl@ceph.osd.2.smithi187.stdout:Jan 24 13:32:24 smithi187 sudo[55343]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-01-24T13:32:24.805 INFO:journalctl@ceph.osd.2.smithi187.stdout:Jan 24 13:32:24 smithi187 sudo[55343]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-24T13:32:24.805 INFO:journalctl@ceph.osd.2.smithi187.stdout:Jan 24 13:32:24 smithi187 sudo[55343]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-24T13:32:24.805 INFO:journalctl@ceph.osd.2.smithi187.stdout:Jan 24 13:32:24 smithi187 sudo[55343]: pam_unix(sudo:session): session closed for user root 2024-01-24T13:32:24.805 INFO:journalctl@ceph.osd.2.smithi187.stdout:Jan 24 13:32:24 smithi187 sudo[55346]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-01-24T13:32:24.805 INFO:journalctl@ceph.osd.2.smithi187.stdout:Jan 24 13:32:24 smithi187 sudo[55346]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-24T13:32:24.805 INFO:journalctl@ceph.osd.2.smithi187.stdout:Jan 24 13:32:24 smithi187 sudo[55346]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-24T13:32:24.805 INFO:journalctl@ceph.osd.2.smithi187.stdout:Jan 24 13:32:24 smithi187 sudo[55346]: pam_unix(sudo:session): session closed for user root 2024-01-24T13:32:24.953 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:24 smithi029 ceph-mon[40651]: osdmap e23: 4 total, 3 up, 4 in 2024-01-24T13:32:24.953 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:24 smithi029 ceph-mon[40651]: mgrmap e16: x(active, since 113s), standbys: z, y 2024-01-24T13:32:24.954 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:24 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T13:32:24.954 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:24 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:24.954 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:24 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:25.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:24 smithi163 ceph-mon[44969]: osdmap e23: 4 total, 3 up, 4 in 2024-01-24T13:32:25.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:24 smithi163 ceph-mon[44969]: mgrmap e16: x(active, since 113s), standbys: z, y 2024-01-24T13:32:25.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:24 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T13:32:25.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:24 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:25.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:24 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:25.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:25 smithi029 sudo[67119]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-01-24T13:32:25.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:25 smithi029 sudo[67119]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-24T13:32:25.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:25 smithi029 sudo[67119]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-24T13:32:25.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:25 smithi029 sudo[67119]: pam_unix(sudo:session): session closed for user root 2024-01-24T13:32:26.078 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:25 smithi029 sudo[67174]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-01-24T13:32:26.078 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:25 smithi029 sudo[67174]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-24T13:32:26.078 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:25 smithi029 sudo[67174]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-24T13:32:26.078 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:25 smithi029 sudo[67174]: pam_unix(sudo:session): session closed for user root 2024-01-24T13:32:26.268 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:26 smithi187 sudo[55349]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-01-24T13:32:26.269 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:26 smithi187 sudo[55349]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-24T13:32:26.269 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:26 smithi187 sudo[55349]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-24T13:32:26.269 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:26 smithi187 sudo[55349]: pam_unix(sudo:session): session closed for user root 2024-01-24T13:32:26.269 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:26 smithi187 sudo[55352]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-01-24T13:32:26.269 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:26 smithi187 sudo[55352]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-24T13:32:26.269 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:26 smithi187 sudo[55352]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-24T13:32:26.269 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:26 smithi187 sudo[55352]: pam_unix(sudo:session): session closed for user root 2024-01-24T13:32:26.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:26 smithi029 ceph-mon[40651]: pgmap v70: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-01-24T13:32:26.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:26 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:26.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:26 smithi029 ceph-mon[40651]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-01-24T13:32:26.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:26 smithi029 ceph-mon[40651]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-01-24T13:32:26.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:26 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-01-24T13:32:26.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:26 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T13:32:26.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:26 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T13:32:26.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:26 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-01-24T13:32:26.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:26 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T13:32:26.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:26 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T13:32:26.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:26 smithi187 ceph-mon[44911]: pgmap v70: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-01-24T13:32:26.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:26 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:26.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:26 smithi187 ceph-mon[44911]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-01-24T13:32:26.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:26 smithi187 ceph-mon[44911]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-01-24T13:32:26.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:26 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-01-24T13:32:26.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:26 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T13:32:26.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:26 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T13:32:26.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:26 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-01-24T13:32:26.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:26 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T13:32:26.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:26 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T13:32:26.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:26 smithi163 ceph-mon[44969]: pgmap v70: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-01-24T13:32:26.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:26 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:26.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:26 smithi163 ceph-mon[44969]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-01-24T13:32:26.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:26 smithi163 ceph-mon[44969]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-01-24T13:32:26.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:26 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-01-24T13:32:26.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:26 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T13:32:26.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:26 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T13:32:26.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:26 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-01-24T13:32:26.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:26 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T13:32:26.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:26 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T13:32:26.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:26 smithi163 sudo[55534]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-01-24T13:32:26.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:26 smithi163 sudo[55534]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-24T13:32:26.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:26 smithi163 sudo[55534]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-24T13:32:26.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:26 smithi163 sudo[55534]: pam_unix(sudo:session): session closed for user root 2024-01-24T13:32:27.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:26 smithi163 sudo[55537]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-01-24T13:32:27.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:26 smithi163 sudo[55537]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-24T13:32:27.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:26 smithi163 sudo[55537]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-24T13:32:27.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:26 smithi163 sudo[55537]: pam_unix(sudo:session): session closed for user root 2024-01-24T13:32:27.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:27 smithi187 ceph-mon[44911]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-01-24T13:32:27.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:27 smithi187 ceph-mon[44911]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-01-24T13:32:27.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:27 smithi187 ceph-mon[44911]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-01-24T13:32:27.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:27 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-01-24T13:32:27.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:27 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T13:32:27.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:27 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T13:32:27.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:27 smithi187 ceph-mon[44911]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-01-24T13:32:27.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:27 smithi187 ceph-mon[44911]: pgmap v71: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-01-24T13:32:27.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:27 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-01-24T13:32:27.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:27 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:27.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:27 smithi187 ceph-mon[44911]: Deploying daemon osd.3 on smithi029 2024-01-24T13:32:27.582 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:27 smithi029 ceph-mon[40651]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-01-24T13:32:27.582 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:27 smithi029 ceph-mon[40651]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-01-24T13:32:27.582 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:27 smithi029 ceph-mon[40651]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-01-24T13:32:27.582 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:27 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-01-24T13:32:27.582 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:27 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T13:32:27.582 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:27 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T13:32:27.582 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:27 smithi029 ceph-mon[40651]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-01-24T13:32:27.583 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:27 smithi029 ceph-mon[40651]: pgmap v71: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-01-24T13:32:27.583 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:27 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-01-24T13:32:27.583 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:27 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:27.583 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:27 smithi029 ceph-mon[40651]: Deploying daemon osd.3 on smithi029 2024-01-24T13:32:27.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:27 smithi163 ceph-mon[44969]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-01-24T13:32:27.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:27 smithi163 ceph-mon[44969]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-01-24T13:32:27.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:27 smithi163 ceph-mon[44969]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-01-24T13:32:27.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:27 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-01-24T13:32:27.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:27 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T13:32:27.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:27 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T13:32:27.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:27 smithi163 ceph-mon[44969]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-01-24T13:32:27.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:27 smithi163 ceph-mon[44969]: pgmap v71: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-01-24T13:32:27.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:27 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-01-24T13:32:27.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:27 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:27.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:27 smithi163 ceph-mon[44969]: Deploying daemon osd.3 on smithi029 2024-01-24T13:32:30.098 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:29 smithi029 ceph-mon[40651]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-01-24T13:32:30.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:29 smithi163 ceph-mon[44969]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-01-24T13:32:30.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:29 smithi187 ceph-mon[44911]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-01-24T13:32:31.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:31 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:31.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:31 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:32:31.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:31 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:31.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:31 smithi163 ceph-mon[44969]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-01-24T13:32:32.001 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:31 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:32.001 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:31 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:32:32.001 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:31 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:32.002 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:31 smithi029 ceph-mon[40651]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-01-24T13:32:32.054 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:31 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:32.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:31 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:32:32.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:31 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:32.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:31 smithi187 ceph-mon[44911]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-01-24T13:32:32.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:32.904+0000 7f9be77fe700 1 -- 172.21.15.29:0/1413489584 <== mgr.14152 v2:172.21.15.29:6800/2498991338 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f9bf8061180 con 0x7f9be00a0490 2024-01-24T13:32:32.904 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 3 on host 'smithi029' 2024-01-24T13:32:32.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:32.906+0000 7f9bfcff4700 1 -- 172.21.15.29:0/1413489584 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f9be00a0490 msgr2=0x7f9be00a2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:32:32.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:32.906+0000 7f9bfcff4700 1 --2- 172.21.15.29:0/1413489584 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f9be00a0490 0x7f9be00a2950 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9bf8100d30 tx=0x7f9be8009260 comp rx=0 tx=0).stop 2024-01-24T13:32:32.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:32.906+0000 7f9bfcff4700 1 -- 172.21.15.29:0/1413489584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf8101870 msgr2=0x7f9bf8102c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:32:32.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:32.906+0000 7f9bfcff4700 1 --2- 172.21.15.29:0/1413489584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf8101870 0x7f9bf8102c00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9bf000c110 tx=0x7f9bf000c420 comp rx=0 tx=0).stop 2024-01-24T13:32:32.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:32.907+0000 7f9bfcff4700 1 -- 172.21.15.29:0/1413489584 shutdown_connections 2024-01-24T13:32:32.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:32.907+0000 7f9bfcff4700 1 --2- 172.21.15.29:0/1413489584 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f9be00a0490 0x7f9be00a2950 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:32:32.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:32.907+0000 7f9bfcff4700 1 --2- 172.21.15.29:0/1413489584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf8101870 0x7f9bf8102c00 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:32:32.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:32.907+0000 7f9bfcff4700 1 --2- 172.21.15.29:0/1413489584 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9bf80ffcd0 0x7f9bf8104070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:32:32.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:32.907+0000 7f9bfcff4700 1 --2- 172.21.15.29:0/1413489584 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9bf8100ed0 0x7f9bf81026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:32:32.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:32.907+0000 7f9bfcff4700 1 -- 172.21.15.29:0/1413489584 >> 172.21.15.29:0/1413489584 conn(0x7f9bf80fb250 msgr2=0x7f9bf80fd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:32:32.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:32.907+0000 7f9bfcff4700 1 -- 172.21.15.29:0/1413489584 shutdown_connections 2024-01-24T13:32:32.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:32:32.907+0000 7f9bfcff4700 1 -- 172.21.15.29:0/1413489584 wait complete. 2024-01-24T13:32:33.287 DEBUG:teuthology.orchestra.run.smithi029:osd.3> sudo journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@osd.3.service 2024-01-24T13:32:33.289 INFO:tasks.cephadm:Deploying osd.4 on smithi163 with /dev/vg_nvme/lv_3... 2024-01-24T13:32:33.289 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-01-24T13:32:33.319 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:33 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:33.319 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:33 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:33.319 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:33 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:33.319 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:33 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:32:33.319 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:33 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:33.319 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:33 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:33.319 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:33 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:33.319 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:33 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:32:33.327 INFO:journalctl@ceph.osd.3.smithi029.stdout:-- Logs begin at Wed 2024-01-24 13:13:42 UTC. -- 2024-01-24T13:32:33.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:33 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:33.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:33 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:33.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:33 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:33.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:33 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:32:33.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:33 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:33.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:33 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:33.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:33 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:33.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:33 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:32:33.522 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.b/config 2024-01-24T13:32:33.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:33 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:33.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:33 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:33.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:33 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:33.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:33 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:32:33.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:33 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:33.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:33 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:33.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:33 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:33.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:33 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:32:34.389 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:34 smithi163 ceph-mon[44969]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-01-24T13:32:34.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:34 smithi187 ceph-mon[44911]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-01-24T13:32:34.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:34 smithi029 ceph-mon[40651]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-01-24T13:32:34.864 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-01-24T13:32:34.864 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-01-24T13:32:34.864 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10+0 records in 2024-01-24T13:32:34.865 INFO:teuthology.orchestra.run.smithi163.stderr:10+0 records out 2024-01-24T13:32:34.865 INFO:teuthology.orchestra.run.smithi163.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112475 s, 932 MB/s 2024-01-24T13:32:34.865 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping successful for: 2024-01-24T13:32:35.099 INFO:journalctl@ceph.osd.3.smithi029.stdout:Jan 24 13:32:34 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-3[67880]: 2024-01-24T13:32:34.655+0000 7f4a6f9f4640 -1 osd.3 0 log_to_monitors true 2024-01-24T13:32:35.183 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:35 smithi163 ceph-mon[44969]: from='osd.3 [v2:172.21.15.29:6810/3417210415,v1:172.21.15.29:6811/3417210415]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-01-24T13:32:35.184 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:35 smithi163 ceph-mon[44969]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-01-24T13:32:35.243 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph orch daemon add osd smithi163:vg_nvme/lv_3 2024-01-24T13:32:35.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:35 smithi029 ceph-mon[40651]: from='osd.3 [v2:172.21.15.29:6810/3417210415,v1:172.21.15.29:6811/3417210415]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-01-24T13:32:35.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:35 smithi029 ceph-mon[40651]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-01-24T13:32:35.460 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.b/config 2024-01-24T13:32:35.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:35 smithi187 ceph-mon[44911]: from='osd.3 [v2:172.21.15.29:6810/3417210415,v1:172.21.15.29:6811/3417210415]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-01-24T13:32:35.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:35 smithi187 ceph-mon[44911]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-01-24T13:32:36.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:36 smithi163 ceph-mon[44969]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-01-24T13:32:36.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:36 smithi163 ceph-mon[44969]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-01-24T13:32:36.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:36 smithi163 ceph-mon[44969]: osdmap e24: 4 total, 3 up, 4 in 2024-01-24T13:32:36.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:36 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T13:32:36.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:36 smithi163 ceph-mon[44969]: from='osd.3 [v2:172.21.15.29:6810/3417210415,v1:172.21.15.29:6811/3417210415]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-01-24T13:32:36.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:36 smithi163 ceph-mon[44969]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-01-24T13:32:36.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:36 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:36.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:36 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:36.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:36 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:32:36.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:36 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:32:36.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:36 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:36.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:36 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:36.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:36 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:32:36.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:36 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:36.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.475+0000 7f68ba18b700 1 -- 172.21.15.163:0/445451963 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f68b4100e20 msgr2=0x7f68b4101240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:32:36.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.475+0000 7f68ba18b700 1 --2- 172.21.15.163:0/445451963 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f68b4100e20 0x7f68b4101240 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f68a80098f0 tx=0x7f68a8009c00 comp rx=0 tx=0).stop 2024-01-24T13:32:36.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.475+0000 7f68ba18b700 1 -- 172.21.15.163:0/445451963 shutdown_connections 2024-01-24T13:32:36.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.475+0000 7f68ba18b700 1 --2- 172.21.15.163:0/445451963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68b4102940 0x7f68b4109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:32:36.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.475+0000 7f68ba18b700 1 --2- 172.21.15.163:0/445451963 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f68b4100e20 0x7f68b4101240 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:32:36.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.475+0000 7f68ba18b700 1 --2- 172.21.15.163:0/445451963 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68b4101f80 0x7f68b4102400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:32:36.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.475+0000 7f68ba18b700 1 -- 172.21.15.163:0/445451963 >> 172.21.15.163:0/445451963 conn(0x7f68b40fc3c0 msgr2=0x7f68b40fe800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:32:36.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.475+0000 7f68ba18b700 1 -- 172.21.15.163:0/445451963 shutdown_connections 2024-01-24T13:32:36.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.475+0000 7f68ba18b700 1 -- 172.21.15.163:0/445451963 wait complete. 2024-01-24T13:32:36.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.476+0000 7f68ba18b700 1 Processor -- start 2024-01-24T13:32:36.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.476+0000 7f68ba18b700 1 -- start start 2024-01-24T13:32:36.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.476+0000 7f68ba18b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68b4100e20 0x7f68b4196ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:32:36.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.476+0000 7f68ba18b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f68b4101f80 0x7f68b4197020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:32:36.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.476+0000 7f68ba18b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68b4102940 0x7f68b419e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:32:36.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.476+0000 7f68ba18b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b419e670 con 0x7f68b4102940 2024-01-24T13:32:36.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.476+0000 7f68ba18b700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b419e7b0 con 0x7f68b4101f80 2024-01-24T13:32:36.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.476+0000 7f68ba18b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b419e940 con 0x7f68b4100e20 2024-01-24T13:32:36.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.476+0000 7f68b2ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f68b4101f80 0x7f68b4197020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:32:36.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.476+0000 7f68b2ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f68b4101f80 0x7f68b4197020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:51186/0 (socket says 172.21.15.163:51186) 2024-01-24T13:32:36.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.476+0000 7f68b2ffd700 1 -- 172.21.15.163:0/3154592917 learned_addr learned my addr 172.21.15.163:0/3154592917 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-01-24T13:32:36.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.477+0000 7f68b37fe700 1 --2- 172.21.15.163:0/3154592917 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68b4100e20 0x7f68b4196ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:32:36.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.477+0000 7f68b3fff700 1 --2- 172.21.15.163:0/3154592917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68b4102940 0x7f68b419e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:32:36.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.477+0000 7f68b2ffd700 1 -- 172.21.15.163:0/3154592917 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68b4100e20 msgr2=0x7f68b4196ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:32:36.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.477+0000 7f68b2ffd700 1 --2- 172.21.15.163:0/3154592917 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68b4100e20 0x7f68b4196ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:32:36.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.477+0000 7f68b2ffd700 1 -- 172.21.15.163:0/3154592917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68b4102940 msgr2=0x7f68b419e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:32:36.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.477+0000 7f68b2ffd700 1 --2- 172.21.15.163:0/3154592917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68b4102940 0x7f68b419e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:32:36.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.477+0000 7f68b2ffd700 1 -- 172.21.15.163:0/3154592917 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a8009520 con 0x7f68b4101f80 2024-01-24T13:32:36.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.477+0000 7f68b3fff700 1 --2- 172.21.15.163:0/3154592917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68b4102940 0x7f68b419e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T13:32:36.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.477+0000 7f68b37fe700 1 --2- 172.21.15.163:0/3154592917 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68b4100e20 0x7f68b4196ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T13:32:36.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.477+0000 7f68b2ffd700 1 --2- 172.21.15.163:0/3154592917 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f68b4101f80 0x7f68b4197020 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f68a400e9c0 tx=0x7f68a400ed80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:32:36.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.477+0000 7f68b0ff9700 1 -- 172.21.15.163:0/3154592917 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a400cbd0 con 0x7f68b4101f80 2024-01-24T13:32:36.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.477+0000 7f68b0ff9700 1 -- 172.21.15.163:0/3154592917 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f68a400cd30 con 0x7f68b4101f80 2024-01-24T13:32:36.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.477+0000 7f68b0ff9700 1 -- 172.21.15.163:0/3154592917 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a4005610 con 0x7f68b4101f80 2024-01-24T13:32:36.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.477+0000 7f68ba18b700 1 -- 172.21.15.163:0/3154592917 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68b419ec20 con 0x7f68b4101f80 2024-01-24T13:32:36.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.477+0000 7f68ba18b700 1 -- 172.21.15.163:0/3154592917 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68b419f0d0 con 0x7f68b4101f80 2024-01-24T13:32:36.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.478+0000 7f68ba18b700 1 -- 172.21.15.163:0/3154592917 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68b410c630 con 0x7f68b4101f80 2024-01-24T13:32:36.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.479+0000 7f68b0ff9700 1 -- 172.21.15.163:0/3154592917 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f68a4004750 con 0x7f68b4101f80 2024-01-24T13:32:36.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.480+0000 7f68b0ff9700 1 --2- 172.21.15.163:0/3154592917 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f689c0a0520 0x7f689c0a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:32:36.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.480+0000 7f68b0ff9700 1 -- 172.21.15.163:0/3154592917 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 3389+0+0 (secure 0 0 0) 0x7f68a4014070 con 0x7f68b4101f80 2024-01-24T13:32:36.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.480+0000 7f68b37fe700 1 --2- 172.21.15.163:0/3154592917 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f689c0a0520 0x7f689c0a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:32:36.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.481+0000 7f68b37fe700 1 --2- 172.21.15.163:0/3154592917 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f689c0a0520 0x7f689c0a29e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f68a800c010 tx=0x7f68a8005b40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:32:36.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.484+0000 7f68b0ff9700 1 -- 172.21.15.163:0/3154592917 <== mon.2 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f68a4095970 con 0x7f68b4101f80 2024-01-24T13:32:36.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:36 smithi187 ceph-mon[44911]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-01-24T13:32:36.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:36 smithi187 ceph-mon[44911]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-01-24T13:32:36.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:36 smithi187 ceph-mon[44911]: osdmap e24: 4 total, 3 up, 4 in 2024-01-24T13:32:36.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:36 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T13:32:36.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:36 smithi187 ceph-mon[44911]: from='osd.3 [v2:172.21.15.29:6810/3417210415,v1:172.21.15.29:6811/3417210415]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-01-24T13:32:36.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:36 smithi187 ceph-mon[44911]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-01-24T13:32:36.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:36 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:36.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:36 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:36.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:36 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:32:36.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:36 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:32:36.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:36 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:36.556 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:36 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:36.556 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:36 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:32:36.556 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:36 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:36.599 INFO:journalctl@ceph.osd.3.smithi029.stdout:Jan 24 13:32:36 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-3[67880]: 2024-01-24T13:32:36.117+0000 7f4a65059700 -1 osd.3 0 waiting for initial osdmap 2024-01-24T13:32:36.599 INFO:journalctl@ceph.osd.3.smithi029.stdout:Jan 24 13:32:36 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-3[67880]: 2024-01-24T13:32:36.124+0000 7f4a5f64b700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-24T13:32:36.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:36 smithi029 ceph-mon[40651]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-01-24T13:32:36.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:36 smithi029 ceph-mon[40651]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-01-24T13:32:36.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:36 smithi029 ceph-mon[40651]: osdmap e24: 4 total, 3 up, 4 in 2024-01-24T13:32:36.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:36 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T13:32:36.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:36 smithi029 ceph-mon[40651]: from='osd.3 [v2:172.21.15.29:6810/3417210415,v1:172.21.15.29:6811/3417210415]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-01-24T13:32:36.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:36 smithi029 ceph-mon[40651]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-01-24T13:32:36.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:36 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:36.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:36 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:36.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:36 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:32:36.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:36 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:32:36.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:36 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:36.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:36 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:36.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:36 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:32:36.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:36 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:36.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:36.676+0000 7f68ba18b700 1 -- 172.21.15.163:0/3154592917 --> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f68b4061180 con 0x7f689c0a0520 2024-01-24T13:32:37.421 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:37 smithi163 ceph-mon[44969]: Detected new or changed devices on smithi029 2024-01-24T13:32:37.421 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:37 smithi163 ceph-mon[44969]: Adjusting osd_memory_target on smithi029 to 8559M 2024-01-24T13:32:37.421 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:37 smithi163 ceph-mon[44969]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-01-24T13:32:37.422 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:37 smithi163 ceph-mon[44969]: osdmap e25: 4 total, 3 up, 4 in 2024-01-24T13:32:37.422 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:37 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T13:32:37.422 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:37 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T13:32:37.422 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:37 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T13:32:37.422 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:37 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T13:32:37.422 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:37 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:37.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:37 smithi187 ceph-mon[44911]: Detected new or changed devices on smithi029 2024-01-24T13:32:37.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:37 smithi187 ceph-mon[44911]: Adjusting osd_memory_target on smithi029 to 8559M 2024-01-24T13:32:37.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:37 smithi187 ceph-mon[44911]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-01-24T13:32:37.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:37 smithi187 ceph-mon[44911]: osdmap e25: 4 total, 3 up, 4 in 2024-01-24T13:32:37.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:37 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T13:32:37.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:37 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T13:32:37.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:37 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T13:32:37.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:37 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T13:32:37.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:37 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:37.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:37 smithi029 ceph-mon[40651]: Detected new or changed devices on smithi029 2024-01-24T13:32:37.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:37 smithi029 ceph-mon[40651]: Adjusting osd_memory_target on smithi029 to 8559M 2024-01-24T13:32:37.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:37 smithi029 ceph-mon[40651]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-01-24T13:32:37.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:37 smithi029 ceph-mon[40651]: osdmap e25: 4 total, 3 up, 4 in 2024-01-24T13:32:37.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:37 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T13:32:37.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:37 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T13:32:37.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:37 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T13:32:37.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:37 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T13:32:37.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:37 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:38.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:38 smithi163 ceph-mon[44969]: purged_snaps scrub starts 2024-01-24T13:32:38.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:38 smithi163 ceph-mon[44969]: purged_snaps scrub ok 2024-01-24T13:32:38.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:38 smithi163 ceph-mon[44969]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:32:38.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:38 smithi163 ceph-mon[44969]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-01-24T13:32:38.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:38 smithi163 ceph-mon[44969]: osd.3 [v2:172.21.15.29:6810/3417210415,v1:172.21.15.29:6811/3417210415] boot 2024-01-24T13:32:38.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:38 smithi163 ceph-mon[44969]: osdmap e26: 4 total, 4 up, 4 in 2024-01-24T13:32:38.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:38 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T13:32:38.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:38 smithi163 ceph-mon[44969]: osdmap e27: 4 total, 4 up, 4 in 2024-01-24T13:32:38.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:38 smithi187 ceph-mon[44911]: purged_snaps scrub starts 2024-01-24T13:32:38.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:38 smithi187 ceph-mon[44911]: purged_snaps scrub ok 2024-01-24T13:32:38.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:38 smithi187 ceph-mon[44911]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:32:38.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:38 smithi187 ceph-mon[44911]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-01-24T13:32:38.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:38 smithi187 ceph-mon[44911]: osd.3 [v2:172.21.15.29:6810/3417210415,v1:172.21.15.29:6811/3417210415] boot 2024-01-24T13:32:38.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:38 smithi187 ceph-mon[44911]: osdmap e26: 4 total, 4 up, 4 in 2024-01-24T13:32:38.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:38 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T13:32:38.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:38 smithi187 ceph-mon[44911]: osdmap e27: 4 total, 4 up, 4 in 2024-01-24T13:32:38.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:38 smithi029 ceph-mon[40651]: purged_snaps scrub starts 2024-01-24T13:32:38.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:38 smithi029 ceph-mon[40651]: purged_snaps scrub ok 2024-01-24T13:32:38.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:38 smithi029 ceph-mon[40651]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:32:38.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:38 smithi029 ceph-mon[40651]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-01-24T13:32:38.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:38 smithi029 ceph-mon[40651]: osd.3 [v2:172.21.15.29:6810/3417210415,v1:172.21.15.29:6811/3417210415] boot 2024-01-24T13:32:38.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:38 smithi029 ceph-mon[40651]: osdmap e26: 4 total, 4 up, 4 in 2024-01-24T13:32:38.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:38 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T13:32:38.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:38 smithi029 ceph-mon[40651]: osdmap e27: 4 total, 4 up, 4 in 2024-01-24T13:32:39.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:39 smithi163 ceph-mon[44969]: from='client.? 172.21.15.163:0/1491059722' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b3a348f-e7dd-4ccb-82a5-2d3f5d101da8"}]: dispatch 2024-01-24T13:32:39.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:39 smithi163 ceph-mon[44969]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b3a348f-e7dd-4ccb-82a5-2d3f5d101da8"}]: dispatch 2024-01-24T13:32:39.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:39 smithi163 ceph-mon[44969]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b3a348f-e7dd-4ccb-82a5-2d3f5d101da8"}]': finished 2024-01-24T13:32:39.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:39 smithi163 ceph-mon[44969]: osdmap e28: 5 total, 4 up, 5 in 2024-01-24T13:32:39.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:39 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T13:32:39.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:39 smithi163 ceph-mon[44969]: from='client.? 172.21.15.163:0/3271548298' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T13:32:39.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:39 smithi187 ceph-mon[44911]: from='client.? 172.21.15.163:0/1491059722' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b3a348f-e7dd-4ccb-82a5-2d3f5d101da8"}]: dispatch 2024-01-24T13:32:39.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:39 smithi187 ceph-mon[44911]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b3a348f-e7dd-4ccb-82a5-2d3f5d101da8"}]: dispatch 2024-01-24T13:32:39.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:39 smithi187 ceph-mon[44911]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b3a348f-e7dd-4ccb-82a5-2d3f5d101da8"}]': finished 2024-01-24T13:32:39.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:39 smithi187 ceph-mon[44911]: osdmap e28: 5 total, 4 up, 5 in 2024-01-24T13:32:39.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:39 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T13:32:39.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:39 smithi187 ceph-mon[44911]: from='client.? 172.21.15.163:0/3271548298' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T13:32:39.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:39 smithi029 ceph-mon[40651]: from='client.? 172.21.15.163:0/1491059722' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b3a348f-e7dd-4ccb-82a5-2d3f5d101da8"}]: dispatch 2024-01-24T13:32:39.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:39 smithi029 ceph-mon[40651]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b3a348f-e7dd-4ccb-82a5-2d3f5d101da8"}]: dispatch 2024-01-24T13:32:39.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:39 smithi029 ceph-mon[40651]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b3a348f-e7dd-4ccb-82a5-2d3f5d101da8"}]': finished 2024-01-24T13:32:39.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:39 smithi029 ceph-mon[40651]: osdmap e28: 5 total, 4 up, 5 in 2024-01-24T13:32:39.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:39 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T13:32:39.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:39 smithi029 ceph-mon[40651]: from='client.? 172.21.15.163:0/3271548298' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T13:32:40.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:40 smithi163 ceph-mon[44969]: pgmap v82: 1 pgs: 1 peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-01-24T13:32:40.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:40 smithi163 ceph-mon[44969]: osdmap e29: 5 total, 4 up, 5 in 2024-01-24T13:32:40.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:40 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T13:32:40.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:40 smithi187 ceph-mon[44911]: pgmap v82: 1 pgs: 1 peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-01-24T13:32:40.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:40 smithi187 ceph-mon[44911]: osdmap e29: 5 total, 4 up, 5 in 2024-01-24T13:32:40.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:40 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T13:32:40.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:40 smithi029 ceph-mon[40651]: pgmap v82: 1 pgs: 1 peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-01-24T13:32:40.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:40 smithi029 ceph-mon[40651]: osdmap e29: 5 total, 4 up, 5 in 2024-01-24T13:32:40.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:40 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T13:32:41.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:41 smithi163 ceph-mon[44969]: osdmap e30: 5 total, 4 up, 5 in 2024-01-24T13:32:41.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:41 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T13:32:41.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:41 smithi163 ceph-mon[44969]: pgmap v85: 1 pgs: 1 peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-01-24T13:32:41.804 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:41 smithi187 ceph-mon[44911]: osdmap e30: 5 total, 4 up, 5 in 2024-01-24T13:32:41.804 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:41 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T13:32:41.804 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:41 smithi187 ceph-mon[44911]: pgmap v85: 1 pgs: 1 peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-01-24T13:32:41.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:41 smithi029 ceph-mon[40651]: osdmap e30: 5 total, 4 up, 5 in 2024-01-24T13:32:41.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:41 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T13:32:41.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:41 smithi029 ceph-mon[40651]: pgmap v85: 1 pgs: 1 peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-01-24T13:32:43.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:43 smithi163 ceph-mon[44969]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-01-24T13:32:44.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:43 smithi029 ceph-mon[40651]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-01-24T13:32:44.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:43 smithi187 ceph-mon[44911]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-01-24T13:32:45.086 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:44 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-01-24T13:32:45.087 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:44 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:45.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:44 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-01-24T13:32:45.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:44 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:45.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:44 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-01-24T13:32:45.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:44 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:46.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:45 smithi029 ceph-mon[40651]: Deploying daemon osd.4 on smithi163 2024-01-24T13:32:46.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:45 smithi029 ceph-mon[40651]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-01-24T13:32:46.106 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:45 smithi163 ceph-mon[44969]: Deploying daemon osd.4 on smithi163 2024-01-24T13:32:46.106 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:45 smithi163 ceph-mon[44969]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-01-24T13:32:46.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:45 smithi187 ceph-mon[44911]: Deploying daemon osd.4 on smithi163 2024-01-24T13:32:46.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:45 smithi187 ceph-mon[44911]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-01-24T13:32:48.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:47 smithi029 ceph-mon[40651]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-01-24T13:32:48.127 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:47 smithi163 ceph-mon[44969]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-01-24T13:32:48.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:47 smithi187 ceph-mon[44911]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-01-24T13:32:49.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:49 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:49.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:49 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:49.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:49 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:32:49.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:49 smithi163 ceph-mon[44969]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-01-24T13:32:49.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:49 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:49.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:49 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:49.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:49 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:32:49.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:49 smithi187 ceph-mon[44911]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-01-24T13:32:49.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:49 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:49.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:49 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:49.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:49 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:32:49.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:49 smithi029 ceph-mon[40651]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-01-24T13:32:50.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:50.432+0000 7f68b0ff9700 1 -- 172.21.15.163:0/3154592917 <== mgr.14152 v2:172.21.15.29:6800/2498991338 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f68b4061180 con 0x7f689c0a0520 2024-01-24T13:32:50.434 INFO:teuthology.orchestra.run.smithi163.stdout:Created osd(s) 4 on host 'smithi163' 2024-01-24T13:32:50.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:50.434+0000 7f68ba18b700 1 -- 172.21.15.163:0/3154592917 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f689c0a0520 msgr2=0x7f689c0a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:32:50.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:50.434+0000 7f68ba18b700 1 --2- 172.21.15.163:0/3154592917 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f689c0a0520 0x7f689c0a29e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f68a800c010 tx=0x7f68a8005b40 comp rx=0 tx=0).stop 2024-01-24T13:32:50.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:50.434+0000 7f68ba18b700 1 -- 172.21.15.163:0/3154592917 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f68b4101f80 msgr2=0x7f68b4197020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:32:50.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:50.434+0000 7f68ba18b700 1 --2- 172.21.15.163:0/3154592917 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f68b4101f80 0x7f68b4197020 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f68a400e9c0 tx=0x7f68a400ed80 comp rx=0 tx=0).stop 2024-01-24T13:32:50.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:50.434+0000 7f68ba18b700 1 -- 172.21.15.163:0/3154592917 shutdown_connections 2024-01-24T13:32:50.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:50.434+0000 7f68ba18b700 1 --2- 172.21.15.163:0/3154592917 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f689c0a0520 0x7f689c0a29e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:32:50.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:50.434+0000 7f68ba18b700 1 --2- 172.21.15.163:0/3154592917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68b4102940 0x7f68b419e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:32:50.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:50.434+0000 7f68ba18b700 1 --2- 172.21.15.163:0/3154592917 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68b4100e20 0x7f68b4196ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:32:50.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:50.434+0000 7f68ba18b700 1 --2- 172.21.15.163:0/3154592917 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f68b4101f80 0x7f68b4197020 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:32:50.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:50.434+0000 7f68ba18b700 1 -- 172.21.15.163:0/3154592917 >> 172.21.15.163:0/3154592917 conn(0x7f68b40fc3c0 msgr2=0x7f68b40fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:32:50.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:50.434+0000 7f68ba18b700 1 -- 172.21.15.163:0/3154592917 shutdown_connections 2024-01-24T13:32:50.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:32:50.434+0000 7f68ba18b700 1 -- 172.21.15.163:0/3154592917 wait complete. 2024-01-24T13:32:50.840 DEBUG:teuthology.orchestra.run.smithi163:osd.4> sudo journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@osd.4.service 2024-01-24T13:32:50.843 INFO:tasks.cephadm:Deploying osd.5 on smithi187 with /dev/vg_nvme/lv_3... 2024-01-24T13:32:50.843 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-01-24T13:32:50.873 INFO:journalctl@ceph.osd.4.smithi163.stdout:-- Logs begin at Wed 2024-01-24 13:14:09 UTC. -- 2024-01-24T13:32:50.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:50 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:50.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:50 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:50.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:50 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:50.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:50 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:32:50.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:50 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:50.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:50 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:50.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:50 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:50.977 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:50 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:32:51.054 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:50 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:51.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:50 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:51.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:50 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:51.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:50 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:32:51.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:50 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:51.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:50 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:51.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:50 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:51.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:50 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:32:51.077 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.c/config 2024-01-24T13:32:51.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:50 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:51.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:50 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:51.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:50 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:51.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:50 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:32:51.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:50 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:51.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:50 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:51.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:50 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:51.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:50 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:32:51.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:51 smithi163 ceph-mon[44969]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-01-24T13:32:52.054 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:51 smithi187 ceph-mon[44911]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-01-24T13:32:52.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:51 smithi029 ceph-mon[40651]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-01-24T13:32:52.349 INFO:teuthology.orchestra.run.smithi187.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-01-24T13:32:52.349 INFO:teuthology.orchestra.run.smithi187.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-01-24T13:32:52.349 INFO:teuthology.orchestra.run.smithi187.stderr: stderr: 10+0 records in 2024-01-24T13:32:52.349 INFO:teuthology.orchestra.run.smithi187.stderr:10+0 records out 2024-01-24T13:32:52.350 INFO:teuthology.orchestra.run.smithi187.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111704 s, 939 MB/s 2024-01-24T13:32:52.350 INFO:teuthology.orchestra.run.smithi187.stderr: stderr: 2024-01-24T13:32:52.350 INFO:teuthology.orchestra.run.smithi187.stderr:--> Zapping successful for: 2024-01-24T13:32:52.482 INFO:journalctl@ceph.osd.4.smithi163.stdout:Jan 24 13:32:52 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-4[58614]: 2024-01-24T13:32:52.200+0000 7f5b7e039640 -1 osd.4 0 log_to_monitors true 2024-01-24T13:32:52.809 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph orch daemon add osd smithi187:vg_nvme/lv_3 2024-01-24T13:32:52.956 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:52 smithi163 ceph-mon[44969]: from='osd.4 [v2:172.21.15.163:6808/3283787183,v1:172.21.15.163:6809/3283787183]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-01-24T13:32:52.956 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:52 smithi163 ceph-mon[44969]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-01-24T13:32:53.027 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.c/config 2024-01-24T13:32:53.054 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:52 smithi187 ceph-mon[44911]: from='osd.4 [v2:172.21.15.163:6808/3283787183,v1:172.21.15.163:6809/3283787183]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-01-24T13:32:53.054 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:52 smithi187 ceph-mon[44911]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-01-24T13:32:53.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:52 smithi029 ceph-mon[40651]: from='osd.4 [v2:172.21.15.163:6808/3283787183,v1:172.21.15.163:6809/3283787183]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-01-24T13:32:53.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:52 smithi029 ceph-mon[40651]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-01-24T13:32:53.988 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:53.985+0000 7fbaa8abf700 1 -- 172.21.15.187:0/3213637823 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbaa4104190 msgr2=0x7fbaa4106580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:32:53.988 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:53.985+0000 7fbaa8abf700 1 --2- 172.21.15.187:0/3213637823 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbaa4104190 0x7fbaa4106580 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fba94009990 tx=0x7fba94009ca0 comp rx=0 tx=0).stop 2024-01-24T13:32:53.989 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:53.985+0000 7fbaa8abf700 1 -- 172.21.15.187:0/3213637823 shutdown_connections 2024-01-24T13:32:53.989 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:53.985+0000 7fbaa8abf700 1 --2- 172.21.15.187:0/3213637823 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbaa4106ac0 0x7fbaa4108f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:32:53.989 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:53.985+0000 7fbaa8abf700 1 --2- 172.21.15.187:0/3213637823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbaa4101860 0x7fbaa4103c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:32:53.989 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:53.985+0000 7fbaa8abf700 1 --2- 172.21.15.187:0/3213637823 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbaa4104190 0x7fbaa4106580 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:32:53.989 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:53.985+0000 7fbaa8abf700 1 -- 172.21.15.187:0/3213637823 >> 172.21.15.187:0/3213637823 conn(0x7fbaa40fb250 msgr2=0x7fbaa40fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:32:53.989 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:53.985+0000 7fbaa8abf700 1 -- 172.21.15.187:0/3213637823 shutdown_connections 2024-01-24T13:32:53.989 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:53.985+0000 7fbaa8abf700 1 -- 172.21.15.187:0/3213637823 wait complete. 2024-01-24T13:32:53.989 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:53.986+0000 7fbaa8abf700 1 Processor -- start 2024-01-24T13:32:53.989 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:53.986+0000 7fbaa8abf700 1 -- start start 2024-01-24T13:32:53.989 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:53.986+0000 7fbaa8abf700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbaa4101860 0x7fbaa4196ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:32:53.989 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:53.986+0000 7fbaa8abf700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbaa4104190 0x7fbaa4196ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:32:53.989 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:53.986+0000 7fbaa8abf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbaa4106ac0 0x7fbaa419e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:32:53.989 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:53.986+0000 7fbaa8abf700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa419e640 con 0x7fbaa4101860 2024-01-24T13:32:53.990 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:53.986+0000 7fbaa8abf700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa419e780 con 0x7fbaa4104190 2024-01-24T13:32:53.990 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:53.986+0000 7fbaa8abf700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa419e910 con 0x7fbaa4106ac0 2024-01-24T13:32:53.990 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:53.986+0000 7fbaa2d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbaa4106ac0 0x7fbaa419e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:32:53.990 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:53.986+0000 7fbaa2d9d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbaa4106ac0 0x7fbaa419e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54404/0 (socket says 172.21.15.187:54404) 2024-01-24T13:32:53.990 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:53.986+0000 7fbaa2d9d700 1 -- 172.21.15.187:0/2860405977 learned_addr learned my addr 172.21.15.187:0/2860405977 (peer_addr_for_me v2:172.21.15.187:0/0) 2024-01-24T13:32:53.990 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:53.986+0000 7fbaa259c700 1 --2- 172.21.15.187:0/2860405977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbaa4101860 0x7fbaa4196ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:32:53.990 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:53.986+0000 7fbaa1d9b700 1 --2- 172.21.15.187:0/2860405977 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbaa4104190 0x7fbaa4196ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:32:53.990 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:53.986+0000 7fbaa2d9d700 1 -- 172.21.15.187:0/2860405977 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbaa4104190 msgr2=0x7fbaa4196ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:32:53.990 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:53.986+0000 7fbaa2d9d700 1 --2- 172.21.15.187:0/2860405977 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbaa4104190 0x7fbaa4196ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:32:53.990 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:53.986+0000 7fbaa2d9d700 1 -- 172.21.15.187:0/2860405977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbaa4101860 msgr2=0x7fbaa4196ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:32:53.990 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:53.986+0000 7fbaa2d9d700 1 --2- 172.21.15.187:0/2860405977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbaa4101860 0x7fbaa4196ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:32:53.990 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:53.986+0000 7fbaa2d9d700 1 -- 172.21.15.187:0/2860405977 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba940095f0 con 0x7fbaa4106ac0 2024-01-24T13:32:53.991 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:53.987+0000 7fbaa2d9d700 1 --2- 172.21.15.187:0/2860405977 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbaa4106ac0 0x7fbaa419e070 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fba9c007ee0 tx=0x7fba9c00d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:32:53.991 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:53.987+0000 7fba937fe700 1 -- 172.21.15.187:0/2860405977 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba9c004db0 con 0x7fbaa4106ac0 2024-01-24T13:32:53.991 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:53.987+0000 7fbaa8abf700 1 -- 172.21.15.187:0/2860405977 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaa419ebf0 con 0x7fbaa4106ac0 2024-01-24T13:32:53.991 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:53.987+0000 7fba937fe700 1 -- 172.21.15.187:0/2860405977 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba9c00f040 con 0x7fbaa4106ac0 2024-01-24T13:32:53.991 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:53.987+0000 7fba937fe700 1 -- 172.21.15.187:0/2860405977 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba9c005920 con 0x7fbaa4106ac0 2024-01-24T13:32:53.991 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:53.987+0000 7fbaa8abf700 1 -- 172.21.15.187:0/2860405977 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaa419f0f0 con 0x7fbaa4106ac0 2024-01-24T13:32:53.991 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:53.988+0000 7fbaa8abf700 1 -- 172.21.15.187:0/2860405977 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaa40fcdd0 con 0x7fbaa4106ac0 2024-01-24T13:32:53.992 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:53.989+0000 7fba937fe700 1 -- 172.21.15.187:0/2860405977 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7fba9c00db20 con 0x7fbaa4106ac0 2024-01-24T13:32:53.992 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:53.989+0000 7fba937fe700 1 --2- 172.21.15.187:0/2860405977 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fba8c0a05a0 0x7fba8c0a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:32:53.992 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:53.990+0000 7fba937fe700 1 -- 172.21.15.187:0/2860405977 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 3821+0+0 (secure 0 0 0) 0x7fba9c0ca1a0 con 0x7fbaa4106ac0 2024-01-24T13:32:53.992 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:53.990+0000 7fbaa259c700 1 --2- 172.21.15.187:0/2860405977 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fba8c0a05a0 0x7fba8c0a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:32:53.992 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:53.990+0000 7fbaa259c700 1 --2- 172.21.15.187:0/2860405977 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fba8c0a05a0 0x7fba8c0a2a60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fba98005950 tx=0x7fba9800a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:32:53.995 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:53.994+0000 7fba937fe700 1 -- 172.21.15.187:0/2860405977 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fba9c094e30 con 0x7fbaa4106ac0 2024-01-24T13:32:54.054 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:53 smithi187 ceph-mon[44911]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-01-24T13:32:54.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:53 smithi187 ceph-mon[44911]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-01-24T13:32:54.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:53 smithi187 ceph-mon[44911]: osdmap e31: 5 total, 4 up, 5 in 2024-01-24T13:32:54.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:53 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T13:32:54.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:53 smithi187 ceph-mon[44911]: from='osd.4 [v2:172.21.15.163:6808/3283787183,v1:172.21.15.163:6809/3283787183]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-01-24T13:32:54.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:53 smithi187 ceph-mon[44911]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-01-24T13:32:54.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:53 smithi187 ceph-mon[44911]: Detected new or changed devices on smithi163 2024-01-24T13:32:54.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:53 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:54.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:53 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:54.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:53 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:32:54.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:53 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:32:54.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:53 smithi187 ceph-mon[44911]: Adjusting osd_memory_target on smithi163 to 8557M 2024-01-24T13:32:54.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:53 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:54.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:53 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:54.056 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:53 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:32:54.056 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:53 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:54.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:53 smithi029 ceph-mon[40651]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-01-24T13:32:54.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:53 smithi029 ceph-mon[40651]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-01-24T13:32:54.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:53 smithi029 ceph-mon[40651]: osdmap e31: 5 total, 4 up, 5 in 2024-01-24T13:32:54.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:53 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T13:32:54.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:53 smithi029 ceph-mon[40651]: from='osd.4 [v2:172.21.15.163:6808/3283787183,v1:172.21.15.163:6809/3283787183]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-01-24T13:32:54.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:53 smithi029 ceph-mon[40651]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-01-24T13:32:54.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:53 smithi029 ceph-mon[40651]: Detected new or changed devices on smithi163 2024-01-24T13:32:54.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:53 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:54.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:53 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:54.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:53 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:32:54.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:53 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:32:54.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:53 smithi029 ceph-mon[40651]: Adjusting osd_memory_target on smithi163 to 8557M 2024-01-24T13:32:54.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:53 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:54.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:53 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:54.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:53 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:32:54.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:53 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:54.189 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:32:54.188+0000 7fbaa8abf700 1 -- 172.21.15.187:0/2860405977 --> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fbaa4061180 con 0x7fba8c0a05a0 2024-01-24T13:32:54.232 INFO:journalctl@ceph.osd.4.smithi163.stdout:Jan 24 13:32:53 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-4[58614]: 2024-01-24T13:32:53.806+0000 7f5b7369e700 -1 osd.4 0 waiting for initial osdmap 2024-01-24T13:32:54.232 INFO:journalctl@ceph.osd.4.smithi163.stdout:Jan 24 13:32:53 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-4[58614]: 2024-01-24T13:32:53.813+0000 7f5b6ec92700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-24T13:32:54.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:53 smithi163 ceph-mon[44969]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-01-24T13:32:54.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:53 smithi163 ceph-mon[44969]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-01-24T13:32:54.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:53 smithi163 ceph-mon[44969]: osdmap e31: 5 total, 4 up, 5 in 2024-01-24T13:32:54.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:53 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T13:32:54.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:53 smithi163 ceph-mon[44969]: from='osd.4 [v2:172.21.15.163:6808/3283787183,v1:172.21.15.163:6809/3283787183]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-01-24T13:32:54.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:53 smithi163 ceph-mon[44969]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-01-24T13:32:54.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:53 smithi163 ceph-mon[44969]: Detected new or changed devices on smithi163 2024-01-24T13:32:54.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:53 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:54.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:53 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:54.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:53 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:32:54.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:53 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:32:54.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:53 smithi163 ceph-mon[44969]: Adjusting osd_memory_target on smithi163 to 8557M 2024-01-24T13:32:54.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:53 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:54.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:53 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:54.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:53 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:32:54.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:53 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:32:55.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:54 smithi029 ceph-mon[40651]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-01-24T13:32:55.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:54 smithi029 ceph-mon[40651]: osdmap e32: 5 total, 4 up, 5 in 2024-01-24T13:32:55.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:54 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T13:32:55.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:54 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T13:32:55.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:54 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T13:32:55.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:54 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T13:32:55.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:54 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:55.165 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:54 smithi187 ceph-mon[44911]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-01-24T13:32:55.165 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:54 smithi187 ceph-mon[44911]: osdmap e32: 5 total, 4 up, 5 in 2024-01-24T13:32:55.166 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:54 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T13:32:55.166 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:54 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T13:32:55.166 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:54 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T13:32:55.166 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:54 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T13:32:55.166 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:54 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:55.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:54 smithi163 ceph-mon[44969]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-01-24T13:32:55.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:54 smithi163 ceph-mon[44969]: osdmap e32: 5 total, 4 up, 5 in 2024-01-24T13:32:55.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:54 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T13:32:55.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:54 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T13:32:55.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:54 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T13:32:55.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:54 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T13:32:55.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:54 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:32:56.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:55 smithi029 ceph-mon[40651]: purged_snaps scrub starts 2024-01-24T13:32:56.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:55 smithi029 ceph-mon[40651]: purged_snaps scrub ok 2024-01-24T13:32:56.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:55 smithi029 ceph-mon[40651]: from='client.24253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:32:56.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:55 smithi029 ceph-mon[40651]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-01-24T13:32:56.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:55 smithi029 ceph-mon[40651]: osd.4 [v2:172.21.15.163:6808/3283787183,v1:172.21.15.163:6809/3283787183] boot 2024-01-24T13:32:56.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:55 smithi029 ceph-mon[40651]: osdmap e33: 5 total, 5 up, 5 in 2024-01-24T13:32:56.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:55 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T13:32:56.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:55 smithi163 ceph-mon[44969]: purged_snaps scrub starts 2024-01-24T13:32:56.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:55 smithi163 ceph-mon[44969]: purged_snaps scrub ok 2024-01-24T13:32:56.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:55 smithi163 ceph-mon[44969]: from='client.24253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:32:56.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:55 smithi163 ceph-mon[44969]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-01-24T13:32:56.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:55 smithi163 ceph-mon[44969]: osd.4 [v2:172.21.15.163:6808/3283787183,v1:172.21.15.163:6809/3283787183] boot 2024-01-24T13:32:56.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:55 smithi163 ceph-mon[44969]: osdmap e33: 5 total, 5 up, 5 in 2024-01-24T13:32:56.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:55 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T13:32:56.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:55 smithi187 ceph-mon[44911]: purged_snaps scrub starts 2024-01-24T13:32:56.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:55 smithi187 ceph-mon[44911]: purged_snaps scrub ok 2024-01-24T13:32:56.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:55 smithi187 ceph-mon[44911]: from='client.24253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:32:56.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:55 smithi187 ceph-mon[44911]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-01-24T13:32:56.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:55 smithi187 ceph-mon[44911]: osd.4 [v2:172.21.15.163:6808/3283787183,v1:172.21.15.163:6809/3283787183] boot 2024-01-24T13:32:56.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:55 smithi187 ceph-mon[44911]: osdmap e33: 5 total, 5 up, 5 in 2024-01-24T13:32:56.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:55 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T13:32:57.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:56 smithi029 ceph-mon[40651]: osdmap e34: 5 total, 5 up, 5 in 2024-01-24T13:32:57.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:56 smithi029 ceph-mon[40651]: from='client.? 172.21.15.187:0/3302547353' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c26c7155-618b-4712-acdb-d4fd6c8b7f66"}]: dispatch 2024-01-24T13:32:57.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:56 smithi029 ceph-mon[40651]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c26c7155-618b-4712-acdb-d4fd6c8b7f66"}]: dispatch 2024-01-24T13:32:57.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:56 smithi029 ceph-mon[40651]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c26c7155-618b-4712-acdb-d4fd6c8b7f66"}]': finished 2024-01-24T13:32:57.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:56 smithi029 ceph-mon[40651]: osdmap e35: 6 total, 5 up, 6 in 2024-01-24T13:32:57.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:56 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-24T13:32:57.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:56 smithi029 ceph-mon[40651]: from='client.? 172.21.15.187:0/254501052' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T13:32:57.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:56 smithi163 ceph-mon[44969]: osdmap e34: 5 total, 5 up, 5 in 2024-01-24T13:32:57.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:56 smithi163 ceph-mon[44969]: from='client.? 172.21.15.187:0/3302547353' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c26c7155-618b-4712-acdb-d4fd6c8b7f66"}]: dispatch 2024-01-24T13:32:57.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:56 smithi163 ceph-mon[44969]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c26c7155-618b-4712-acdb-d4fd6c8b7f66"}]: dispatch 2024-01-24T13:32:57.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:56 smithi163 ceph-mon[44969]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c26c7155-618b-4712-acdb-d4fd6c8b7f66"}]': finished 2024-01-24T13:32:57.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:56 smithi163 ceph-mon[44969]: osdmap e35: 6 total, 5 up, 6 in 2024-01-24T13:32:57.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:56 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-24T13:32:57.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:56 smithi163 ceph-mon[44969]: from='client.? 172.21.15.187:0/254501052' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T13:32:57.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:56 smithi187 ceph-mon[44911]: osdmap e34: 5 total, 5 up, 5 in 2024-01-24T13:32:57.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:56 smithi187 ceph-mon[44911]: from='client.? 172.21.15.187:0/3302547353' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c26c7155-618b-4712-acdb-d4fd6c8b7f66"}]: dispatch 2024-01-24T13:32:57.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:56 smithi187 ceph-mon[44911]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c26c7155-618b-4712-acdb-d4fd6c8b7f66"}]: dispatch 2024-01-24T13:32:57.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:56 smithi187 ceph-mon[44911]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c26c7155-618b-4712-acdb-d4fd6c8b7f66"}]': finished 2024-01-24T13:32:57.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:56 smithi187 ceph-mon[44911]: osdmap e35: 6 total, 5 up, 6 in 2024-01-24T13:32:57.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:56 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-24T13:32:57.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:56 smithi187 ceph-mon[44911]: from='client.? 172.21.15.187:0/254501052' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T13:32:58.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:57 smithi163 ceph-mon[44969]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-01-24T13:32:58.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:57 smithi187 ceph-mon[44911]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-01-24T13:32:58.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:57 smithi029 ceph-mon[40651]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-01-24T13:33:00.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:32:59 smithi163 ceph-mon[44969]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-01-24T13:33:00.279 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:32:59 smithi187 ceph-mon[44911]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-01-24T13:33:00.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:32:59 smithi029 ceph-mon[40651]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-01-24T13:33:02.054 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:01 smithi187 ceph-mon[44911]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-01-24T13:33:02.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:01 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-01-24T13:33:02.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:01 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:33:02.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:01 smithi163 ceph-mon[44969]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-01-24T13:33:02.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:01 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-01-24T13:33:02.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:01 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:33:02.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:01 smithi029 ceph-mon[40651]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-01-24T13:33:02.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:01 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-01-24T13:33:02.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:01 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:33:03.210 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:02 smithi187 ceph-mon[44911]: Deploying daemon osd.5 on smithi187 2024-01-24T13:33:03.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:02 smithi163 ceph-mon[44969]: Deploying daemon osd.5 on smithi187 2024-01-24T13:33:03.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:02 smithi029 ceph-mon[40651]: Deploying daemon osd.5 on smithi187 2024-01-24T13:33:04.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:03 smithi187 ceph-mon[44911]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-01-24T13:33:04.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:03 smithi163 ceph-mon[44969]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-01-24T13:33:04.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:03 smithi029 ceph-mon[40651]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-01-24T13:33:06.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:05 smithi187 ceph-mon[44911]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-01-24T13:33:06.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:05 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:06.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:05 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:06.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:05 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:33:06.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:05 smithi163 ceph-mon[44969]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-01-24T13:33:06.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:05 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:06.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:05 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:06.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:05 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:33:06.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:05 smithi029 ceph-mon[40651]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-01-24T13:33:06.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:05 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:06.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:05 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:06.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:05 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:33:07.496 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:07.496+0000 7fba937fe700 1 -- 172.21.15.187:0/2860405977 <== mgr.14152 v2:172.21.15.29:6800/2498991338 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fbaa4061180 con 0x7fba8c0a05a0 2024-01-24T13:33:07.497 INFO:teuthology.orchestra.run.smithi187.stdout:Created osd(s) 5 on host 'smithi187' 2024-01-24T13:33:07.499 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:07.499+0000 7fbaa8abf700 1 -- 172.21.15.187:0/2860405977 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fba8c0a05a0 msgr2=0x7fba8c0a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:33:07.499 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:07.499+0000 7fbaa8abf700 1 --2- 172.21.15.187:0/2860405977 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fba8c0a05a0 0x7fba8c0a2a60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fba98005950 tx=0x7fba9800a300 comp rx=0 tx=0).stop 2024-01-24T13:33:07.499 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:07.499+0000 7fbaa8abf700 1 -- 172.21.15.187:0/2860405977 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbaa4106ac0 msgr2=0x7fbaa419e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:33:07.499 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:07.499+0000 7fbaa8abf700 1 --2- 172.21.15.187:0/2860405977 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbaa4106ac0 0x7fbaa419e070 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fba9c007ee0 tx=0x7fba9c00d3b0 comp rx=0 tx=0).stop 2024-01-24T13:33:07.499 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:07.499+0000 7fbaa8abf700 1 -- 172.21.15.187:0/2860405977 shutdown_connections 2024-01-24T13:33:07.499 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:07.499+0000 7fbaa8abf700 1 --2- 172.21.15.187:0/2860405977 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbaa4106ac0 0x7fbaa419e070 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:33:07.499 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:07.499+0000 7fbaa8abf700 1 --2- 172.21.15.187:0/2860405977 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fba8c0a05a0 0x7fba8c0a2a60 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:33:07.499 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:07.499+0000 7fbaa8abf700 1 --2- 172.21.15.187:0/2860405977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbaa4101860 0x7fbaa4196ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:33:07.499 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:07.499+0000 7fbaa8abf700 1 --2- 172.21.15.187:0/2860405977 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbaa4104190 0x7fbaa4196ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:33:07.499 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:07.499+0000 7fbaa8abf700 1 -- 172.21.15.187:0/2860405977 >> 172.21.15.187:0/2860405977 conn(0x7fbaa40fb250 msgr2=0x7fbaa41024d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:33:07.499 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:07.499+0000 7fbaa8abf700 1 -- 172.21.15.187:0/2860405977 shutdown_connections 2024-01-24T13:33:07.499 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:07.499+0000 7fbaa8abf700 1 -- 172.21.15.187:0/2860405977 wait complete. 2024-01-24T13:33:07.663 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:07 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:07.663 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:07 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:07.663 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:07 smithi187 ceph-mon[44911]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-01-24T13:33:07.664 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:07 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:33:07.664 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:07 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:33:07.664 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:07 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:07.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:07 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:07.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:07 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:07.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:07 smithi163 ceph-mon[44969]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-01-24T13:33:07.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:07 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:33:07.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:07 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:33:07.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:07 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:07.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:07 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:07.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:07 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:07.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:07 smithi029 ceph-mon[40651]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-01-24T13:33:07.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:07 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:33:07.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:07 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:33:07.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:07 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:07.895 DEBUG:teuthology.orchestra.run.smithi187:osd.5> sudo journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@osd.5.service 2024-01-24T13:33:07.897 INFO:tasks.cephadm:Deploying osd.6 on smithi029 with /dev/vg_nvme/lv_2... 2024-01-24T13:33:07.897 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-01-24T13:33:07.928 INFO:journalctl@ceph.osd.5.smithi187.stdout:-- Logs begin at Wed 2024-01-24 13:13:23 UTC. -- 2024-01-24T13:33:08.152 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T13:33:08.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:08 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:08.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:08 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:08.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:08 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:33:08.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:08 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:08.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:08 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:08.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:08 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:33:08.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:08 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:08.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:08 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:08.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:08 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:33:09.499 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-01-24T13:33:09.499 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-01-24T13:33:09.499 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10+0 records in 2024-01-24T13:33:09.499 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2024-01-24T13:33:09.499 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00899554 s, 1.2 GB/s 2024-01-24T13:33:09.500 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping successful for: 2024-01-24T13:33:09.655 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:09 smithi187 ceph-mon[44911]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-01-24T13:33:09.655 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:09 smithi187 ceph-mon[44911]: from='osd.5 [v2:172.21.15.187:6808/2259228817,v1:172.21.15.187:6809/2259228817]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-01-24T13:33:09.655 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:09 smithi187 ceph-mon[44911]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-01-24T13:33:09.655 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jan 24 13:33:09 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-5[58508]: 2024-01-24T13:33:09.313+0000 7fdb06b8c640 -1 osd.5 0 log_to_monitors true 2024-01-24T13:33:09.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:09 smithi163 ceph-mon[44969]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-01-24T13:33:09.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:09 smithi163 ceph-mon[44969]: from='osd.5 [v2:172.21.15.187:6808/2259228817,v1:172.21.15.187:6809/2259228817]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-01-24T13:33:09.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:09 smithi163 ceph-mon[44969]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-01-24T13:33:09.790 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:09 smithi029 ceph-mon[40651]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-01-24T13:33:09.790 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:09 smithi029 ceph-mon[40651]: from='osd.5 [v2:172.21.15.187:6808/2259228817,v1:172.21.15.187:6809/2259228817]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-01-24T13:33:09.790 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:09 smithi029 ceph-mon[40651]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-01-24T13:33:09.837 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph orch daemon add osd smithi029:vg_nvme/lv_2 2024-01-24T13:33:10.099 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T13:33:10.472 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:10 smithi029 ceph-mon[40651]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-01-24T13:33:10.472 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:10 smithi029 ceph-mon[40651]: osdmap e36: 6 total, 5 up, 6 in 2024-01-24T13:33:10.472 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:10 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-24T13:33:10.472 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:10 smithi029 ceph-mon[40651]: from='osd.5 [v2:172.21.15.187:6808/2259228817,v1:172.21.15.187:6809/2259228817]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-01-24T13:33:10.472 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:10 smithi029 ceph-mon[40651]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-01-24T13:33:10.472 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:10 smithi029 ceph-mon[40651]: Detected new or changed devices on smithi187 2024-01-24T13:33:10.472 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:10 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:10.472 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:10 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:10.472 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:10 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:33:10.472 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:10 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:33:10.472 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:10 smithi029 ceph-mon[40651]: Adjusting osd_memory_target on smithi187 to 8556M 2024-01-24T13:33:10.473 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:10 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:10.473 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:10 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:33:10.473 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:10 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:33:10.473 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:10 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:10.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:10 smithi163 ceph-mon[44969]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-01-24T13:33:10.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:10 smithi163 ceph-mon[44969]: osdmap e36: 6 total, 5 up, 6 in 2024-01-24T13:33:10.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:10 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-24T13:33:10.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:10 smithi163 ceph-mon[44969]: from='osd.5 [v2:172.21.15.187:6808/2259228817,v1:172.21.15.187:6809/2259228817]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-01-24T13:33:10.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:10 smithi163 ceph-mon[44969]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-01-24T13:33:10.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:10 smithi163 ceph-mon[44969]: Detected new or changed devices on smithi187 2024-01-24T13:33:10.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:10 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:10.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:10 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:10.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:10 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:33:10.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:10 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:33:10.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:10 smithi163 ceph-mon[44969]: Adjusting osd_memory_target on smithi187 to 8556M 2024-01-24T13:33:10.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:10 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:10.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:10 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:33:10.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:10 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:33:10.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:10 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:10.804 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jan 24 13:33:10 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-5[58508]: 2024-01-24T13:33:10.475+0000 7fdafc1f1700 -1 osd.5 0 waiting for initial osdmap 2024-01-24T13:33:10.805 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jan 24 13:33:10 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-5[58508]: 2024-01-24T13:33:10.482+0000 7fdaf3fde700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-24T13:33:10.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:10 smithi187 ceph-mon[44911]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-01-24T13:33:10.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:10 smithi187 ceph-mon[44911]: osdmap e36: 6 total, 5 up, 6 in 2024-01-24T13:33:10.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:10 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-24T13:33:10.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:10 smithi187 ceph-mon[44911]: from='osd.5 [v2:172.21.15.187:6808/2259228817,v1:172.21.15.187:6809/2259228817]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-01-24T13:33:10.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:10 smithi187 ceph-mon[44911]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-01-24T13:33:10.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:10 smithi187 ceph-mon[44911]: Detected new or changed devices on smithi187 2024-01-24T13:33:10.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:10 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:10.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:10 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:10.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:10 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:33:10.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:10 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:33:10.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:10 smithi187 ceph-mon[44911]: Adjusting osd_memory_target on smithi187 to 8556M 2024-01-24T13:33:10.806 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:10 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:10.806 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:10 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:33:10.806 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:10 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:33:10.806 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:10 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:11.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.099+0000 7ffb8e9cf700 1 -- 172.21.15.29:0/2295374744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb88101fd0 msgr2=0x7ffb88102430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:33:11.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.099+0000 7ffb8e9cf700 1 --2- 172.21.15.29:0/2295374744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb88101fd0 0x7ffb88102430 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ffb780099c0 tx=0x7ffb78009cd0 comp rx=0 tx=0).stop 2024-01-24T13:33:11.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.099+0000 7ffb8e9cf700 1 -- 172.21.15.29:0/2295374744 shutdown_connections 2024-01-24T13:33:11.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.099+0000 7ffb8e9cf700 1 --2- 172.21.15.29:0/2295374744 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffb88102970 0x7ffb88109220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:33:11.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.099+0000 7ffb8e9cf700 1 --2- 172.21.15.29:0/2295374744 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb88100dd0 0x7ffb881011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:33:11.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.099+0000 7ffb8e9cf700 1 --2- 172.21.15.29:0/2295374744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb88101fd0 0x7ffb88102430 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:33:11.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.099+0000 7ffb8e9cf700 1 -- 172.21.15.29:0/2295374744 >> 172.21.15.29:0/2295374744 conn(0x7ffb880fc330 msgr2=0x7ffb880fe7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:33:11.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.099+0000 7ffb8e9cf700 1 -- 172.21.15.29:0/2295374744 shutdown_connections 2024-01-24T13:33:11.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.099+0000 7ffb8e9cf700 1 -- 172.21.15.29:0/2295374744 wait complete. 2024-01-24T13:33:11.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.100+0000 7ffb8e9cf700 1 Processor -- start 2024-01-24T13:33:11.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.100+0000 7ffb8e9cf700 1 -- start start 2024-01-24T13:33:11.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.100+0000 7ffb8e9cf700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffb88100dd0 0x7ffb88196a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:33:11.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.100+0000 7ffb8e9cf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb88101fd0 0x7ffb88196fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:33:11.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.100+0000 7ffb8e9cf700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb88102970 0x7ffb8819e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:33:11.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.100+0000 7ffb8e9cf700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8819e620 con 0x7ffb88102970 2024-01-24T13:33:11.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.100+0000 7ffb8e9cf700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8819e760 con 0x7ffb88100dd0 2024-01-24T13:33:11.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.100+0000 7ffb8e9cf700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8819e8f0 con 0x7ffb88101fd0 2024-01-24T13:33:11.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.100+0000 7ffb8cf6c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb88102970 0x7ffb8819e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:33:11.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.100+0000 7ffb8cf6c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb88102970 0x7ffb8819e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42204/0 (socket says 172.21.15.29:42204) 2024-01-24T13:33:11.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.100+0000 7ffb8cf6c700 1 -- 172.21.15.29:0/3193847790 learned_addr learned my addr 172.21.15.29:0/3193847790 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:33:11.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.100+0000 7ffb87fff700 1 --2- 172.21.15.29:0/3193847790 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffb88100dd0 0x7ffb88196a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:33:11.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.100+0000 7ffb877fe700 1 --2- 172.21.15.29:0/3193847790 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb88101fd0 0x7ffb88196fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:33:11.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.101+0000 7ffb8cf6c700 1 -- 172.21.15.29:0/3193847790 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb88101fd0 msgr2=0x7ffb88196fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:33:11.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.101+0000 7ffb8cf6c700 1 --2- 172.21.15.29:0/3193847790 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb88101fd0 0x7ffb88196fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:33:11.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.101+0000 7ffb8cf6c700 1 -- 172.21.15.29:0/3193847790 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffb88100dd0 msgr2=0x7ffb88196a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:33:11.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.101+0000 7ffb8cf6c700 1 --2- 172.21.15.29:0/3193847790 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffb88100dd0 0x7ffb88196a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:33:11.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.101+0000 7ffb8cf6c700 1 -- 172.21.15.29:0/3193847790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb780095f0 con 0x7ffb88102970 2024-01-24T13:33:11.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.101+0000 7ffb87fff700 1 --2- 172.21.15.29:0/3193847790 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffb88100dd0 0x7ffb88196a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T13:33:11.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.101+0000 7ffb8cf6c700 1 --2- 172.21.15.29:0/3193847790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb88102970 0x7ffb8819e050 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ffb80007f10 tx=0x7ffb8000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:33:11.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.101+0000 7ffb857fa700 1 -- 172.21.15.29:0/3193847790 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb80017070 con 0x7ffb88102970 2024-01-24T13:33:11.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.101+0000 7ffb857fa700 1 -- 172.21.15.29:0/3193847790 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb8000f040 con 0x7ffb88102970 2024-01-24T13:33:11.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.101+0000 7ffb8e9cf700 1 -- 172.21.15.29:0/3193847790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb8819ebd0 con 0x7ffb88102970 2024-01-24T13:33:11.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.101+0000 7ffb857fa700 1 -- 172.21.15.29:0/3193847790 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb80005740 con 0x7ffb88102970 2024-01-24T13:33:11.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.101+0000 7ffb8e9cf700 1 -- 172.21.15.29:0/3193847790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb8819f080 con 0x7ffb88102970 2024-01-24T13:33:11.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.102+0000 7ffb8e9cf700 1 -- 172.21.15.29:0/3193847790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb8810c5d0 con 0x7ffb88102970 2024-01-24T13:33:11.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.103+0000 7ffb857fa700 1 -- 172.21.15.29:0/3193847790 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7ffb800058a0 con 0x7ffb88102970 2024-01-24T13:33:11.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.104+0000 7ffb857fa700 1 --2- 172.21.15.29:0/3193847790 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7ffb740a0520 0x7ffb740a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:33:11.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.104+0000 7ffb87fff700 1 --2- 172.21.15.29:0/3193847790 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7ffb740a0520 0x7ffb740a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:33:11.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.104+0000 7ffb857fa700 1 -- 172.21.15.29:0/3193847790 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4566+0+0 (secure 0 0 0) 0x7ffb800caa10 con 0x7ffb88102970 2024-01-24T13:33:11.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.105+0000 7ffb87fff700 1 --2- 172.21.15.29:0/3193847790 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7ffb740a0520 0x7ffb740a29e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ffb70005950 tx=0x7ffb700058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:33:11.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.108+0000 7ffb857fa700 1 -- 172.21.15.29:0/3193847790 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7ffb80095430 con 0x7ffb88102970 2024-01-24T13:33:11.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:11.326+0000 7ffb8e9cf700 1 -- 172.21.15.29:0/3193847790 --> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7ffb88061180 con 0x7ffb740a0520 2024-01-24T13:33:11.731 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:11 smithi029 ceph-mon[40651]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-01-24T13:33:11.731 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:11 smithi029 ceph-mon[40651]: osdmap e37: 6 total, 5 up, 6 in 2024-01-24T13:33:11.731 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:11 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-24T13:33:11.731 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:11 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-24T13:33:11.731 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:11 smithi029 ceph-mon[40651]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-01-24T13:33:11.731 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:11 smithi029 ceph-mon[40651]: osd.5 [v2:172.21.15.187:6808/2259228817,v1:172.21.15.187:6809/2259228817] boot 2024-01-24T13:33:11.731 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:11 smithi029 ceph-mon[40651]: osdmap e38: 6 total, 6 up, 6 in 2024-01-24T13:33:11.731 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:11 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-24T13:33:11.731 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:11 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T13:33:11.731 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:11 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T13:33:11.731 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:11 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:33:11.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:11 smithi163 ceph-mon[44969]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-01-24T13:33:11.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:11 smithi163 ceph-mon[44969]: osdmap e37: 6 total, 5 up, 6 in 2024-01-24T13:33:11.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:11 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-24T13:33:11.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:11 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-24T13:33:11.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:11 smithi163 ceph-mon[44969]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-01-24T13:33:11.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:11 smithi163 ceph-mon[44969]: osd.5 [v2:172.21.15.187:6808/2259228817,v1:172.21.15.187:6809/2259228817] boot 2024-01-24T13:33:11.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:11 smithi163 ceph-mon[44969]: osdmap e38: 6 total, 6 up, 6 in 2024-01-24T13:33:11.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:11 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-24T13:33:11.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:11 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T13:33:11.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:11 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T13:33:11.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:11 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:33:11.804 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:11 smithi187 ceph-mon[44911]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-01-24T13:33:11.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:11 smithi187 ceph-mon[44911]: osdmap e37: 6 total, 5 up, 6 in 2024-01-24T13:33:11.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:11 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-24T13:33:11.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:11 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-24T13:33:11.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:11 smithi187 ceph-mon[44911]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-01-24T13:33:11.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:11 smithi187 ceph-mon[44911]: osd.5 [v2:172.21.15.187:6808/2259228817,v1:172.21.15.187:6809/2259228817] boot 2024-01-24T13:33:11.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:11 smithi187 ceph-mon[44911]: osdmap e38: 6 total, 6 up, 6 in 2024-01-24T13:33:11.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:11 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-24T13:33:11.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:11 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T13:33:11.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:11 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T13:33:11.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:11 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:33:12.804 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:12 smithi029 ceph-mon[40651]: purged_snaps scrub starts 2024-01-24T13:33:12.804 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:12 smithi029 ceph-mon[40651]: purged_snaps scrub ok 2024-01-24T13:33:12.804 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:12 smithi029 ceph-mon[40651]: from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:33:12.804 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:12 smithi029 ceph-mon[40651]: osdmap e39: 6 total, 6 up, 6 in 2024-01-24T13:33:13.054 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:12 smithi187 ceph-mon[44911]: purged_snaps scrub starts 2024-01-24T13:33:13.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:12 smithi187 ceph-mon[44911]: purged_snaps scrub ok 2024-01-24T13:33:13.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:12 smithi187 ceph-mon[44911]: from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:33:13.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:12 smithi187 ceph-mon[44911]: osdmap e39: 6 total, 6 up, 6 in 2024-01-24T13:33:13.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:12 smithi163 ceph-mon[44969]: purged_snaps scrub starts 2024-01-24T13:33:13.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:12 smithi163 ceph-mon[44969]: purged_snaps scrub ok 2024-01-24T13:33:13.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:12 smithi163 ceph-mon[44969]: from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:33:13.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:12 smithi163 ceph-mon[44969]: osdmap e39: 6 total, 6 up, 6 in 2024-01-24T13:33:14.054 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:13 smithi187 ceph-mon[44911]: pgmap v110: 1 pgs: 1 remapped+peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2024-01-24T13:33:14.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:13 smithi187 ceph-mon[44911]: osdmap e40: 6 total, 6 up, 6 in 2024-01-24T13:33:14.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:13 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/532720200' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "042ac0b4-1fe1-444f-bf51-ad89613a2185"}]: dispatch 2024-01-24T13:33:14.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:13 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/532720200' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "042ac0b4-1fe1-444f-bf51-ad89613a2185"}]': finished 2024-01-24T13:33:14.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:13 smithi187 ceph-mon[44911]: osdmap e41: 7 total, 6 up, 7 in 2024-01-24T13:33:14.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:13 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-01-24T13:33:14.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:13 smithi029 ceph-mon[40651]: pgmap v110: 1 pgs: 1 remapped+peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2024-01-24T13:33:14.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:13 smithi029 ceph-mon[40651]: osdmap e40: 6 total, 6 up, 6 in 2024-01-24T13:33:14.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:13 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/532720200' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "042ac0b4-1fe1-444f-bf51-ad89613a2185"}]: dispatch 2024-01-24T13:33:14.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:13 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/532720200' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "042ac0b4-1fe1-444f-bf51-ad89613a2185"}]': finished 2024-01-24T13:33:14.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:13 smithi029 ceph-mon[40651]: osdmap e41: 7 total, 6 up, 7 in 2024-01-24T13:33:14.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:13 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-01-24T13:33:14.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:13 smithi163 ceph-mon[44969]: pgmap v110: 1 pgs: 1 remapped+peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2024-01-24T13:33:14.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:13 smithi163 ceph-mon[44969]: osdmap e40: 6 total, 6 up, 6 in 2024-01-24T13:33:14.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:13 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/532720200' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "042ac0b4-1fe1-444f-bf51-ad89613a2185"}]: dispatch 2024-01-24T13:33:14.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:13 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/532720200' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "042ac0b4-1fe1-444f-bf51-ad89613a2185"}]': finished 2024-01-24T13:33:14.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:13 smithi163 ceph-mon[44969]: osdmap e41: 7 total, 6 up, 7 in 2024-01-24T13:33:14.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:13 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-01-24T13:33:15.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:14 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/1475933321' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T13:33:15.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:14 smithi029 ceph-mon[40651]: osdmap e42: 7 total, 6 up, 7 in 2024-01-24T13:33:15.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:14 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-01-24T13:33:15.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:14 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/1475933321' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T13:33:15.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:14 smithi163 ceph-mon[44969]: osdmap e42: 7 total, 6 up, 7 in 2024-01-24T13:33:15.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:14 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-01-24T13:33:15.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:14 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/1475933321' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T13:33:15.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:14 smithi187 ceph-mon[44911]: osdmap e42: 7 total, 6 up, 7 in 2024-01-24T13:33:15.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:14 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-01-24T13:33:16.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:15 smithi029 ceph-mon[40651]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2024-01-24T13:33:16.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:15 smithi163 ceph-mon[44969]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2024-01-24T13:33:16.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:15 smithi187 ceph-mon[44911]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2024-01-24T13:33:18.043 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:17 smithi029 ceph-mon[40651]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-01-24T13:33:18.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:17 smithi163 ceph-mon[44969]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-01-24T13:33:18.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:17 smithi187 ceph-mon[44911]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-01-24T13:33:19.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:19 smithi029 ceph-mon[40651]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-01-24T13:33:19.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:19 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-01-24T13:33:19.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:19 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:33:19.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:19 smithi029 ceph-mon[40651]: Deploying daemon osd.6 on smithi029 2024-01-24T13:33:20.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:19 smithi163 ceph-mon[44969]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-01-24T13:33:20.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:19 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-01-24T13:33:20.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:19 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:33:20.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:19 smithi163 ceph-mon[44969]: Deploying daemon osd.6 on smithi029 2024-01-24T13:33:20.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:19 smithi187 ceph-mon[44911]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-01-24T13:33:20.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:19 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-01-24T13:33:20.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:19 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:33:20.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:19 smithi187 ceph-mon[44911]: Deploying daemon osd.6 on smithi029 2024-01-24T13:33:22.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:21 smithi029 ceph-mon[40651]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-01-24T13:33:22.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:21 smithi163 ceph-mon[44969]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-01-24T13:33:22.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:21 smithi187 ceph-mon[44911]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-01-24T13:33:24.025 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:23 smithi029 ceph-mon[40651]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-01-24T13:33:24.056 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:23 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:24.056 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:23 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:24.056 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:23 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:33:24.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:23 smithi163 ceph-mon[44969]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-01-24T13:33:24.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:23 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:24.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:23 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:24.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:23 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:33:24.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:23 smithi187 ceph-mon[44911]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-01-24T13:33:24.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:23 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:24.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:23 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:24.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:23 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:33:25.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:25.319+0000 7ffb857fa700 1 -- 172.21.15.29:0/3193847790 <== mgr.14152 v2:172.21.15.29:6800/2498991338 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ffb88061180 con 0x7ffb740a0520 2024-01-24T13:33:25.319 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 6 on host 'smithi029' 2024-01-24T13:33:25.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:25.322+0000 7ffb8e9cf700 1 -- 172.21.15.29:0/3193847790 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7ffb740a0520 msgr2=0x7ffb740a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:33:25.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:25.322+0000 7ffb8e9cf700 1 --2- 172.21.15.29:0/3193847790 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7ffb740a0520 0x7ffb740a29e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ffb70005950 tx=0x7ffb700058e0 comp rx=0 tx=0).stop 2024-01-24T13:33:25.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:25.322+0000 7ffb8e9cf700 1 -- 172.21.15.29:0/3193847790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb88102970 msgr2=0x7ffb8819e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:33:25.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:25.322+0000 7ffb8e9cf700 1 --2- 172.21.15.29:0/3193847790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb88102970 0x7ffb8819e050 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ffb80007f10 tx=0x7ffb8000d3b0 comp rx=0 tx=0).stop 2024-01-24T13:33:25.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:25.323+0000 7ffb8e9cf700 1 -- 172.21.15.29:0/3193847790 shutdown_connections 2024-01-24T13:33:25.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:25.323+0000 7ffb8e9cf700 1 --2- 172.21.15.29:0/3193847790 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7ffb740a0520 0x7ffb740a29e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:33:25.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:25.323+0000 7ffb8e9cf700 1 --2- 172.21.15.29:0/3193847790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb88102970 0x7ffb8819e050 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:33:25.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:25.323+0000 7ffb8e9cf700 1 --2- 172.21.15.29:0/3193847790 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffb88100dd0 0x7ffb88196a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:33:25.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:25.323+0000 7ffb8e9cf700 1 --2- 172.21.15.29:0/3193847790 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb88101fd0 0x7ffb88196fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:33:25.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:25.323+0000 7ffb8e9cf700 1 -- 172.21.15.29:0/3193847790 >> 172.21.15.29:0/3193847790 conn(0x7ffb880fc330 msgr2=0x7ffb880fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:33:25.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:25.323+0000 7ffb8e9cf700 1 -- 172.21.15.29:0/3193847790 shutdown_connections 2024-01-24T13:33:25.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:33:25.323+0000 7ffb8e9cf700 1 -- 172.21.15.29:0/3193847790 wait complete. 2024-01-24T13:33:25.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:25 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:25.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:25 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:25.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:25 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:33:25.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:25 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:33:25.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:25 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:25.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:25 smithi029 ceph-mon[40651]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-01-24T13:33:25.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:25 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:25.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:25 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:25.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:25 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:33:25.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:25 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:25.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:25 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:25.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:25 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:33:25.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:25 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:33:25.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:25 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:25.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:25 smithi163 ceph-mon[44969]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-01-24T13:33:25.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:25 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:25.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:25 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:25.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:25 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:33:25.804 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:25 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:25.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:25 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:25.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:25 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:33:25.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:25 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:33:25.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:25 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:25.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:25 smithi187 ceph-mon[44911]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-01-24T13:33:25.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:25 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:25.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:25 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:25.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:25 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:33:26.030 DEBUG:teuthology.orchestra.run.smithi029:osd.6> sudo journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@osd.6.service 2024-01-24T13:33:26.033 INFO:tasks.cephadm:Deploying osd.7 on smithi163 with /dev/vg_nvme/lv_2... 2024-01-24T13:33:26.033 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-01-24T13:33:26.070 INFO:journalctl@ceph.osd.6.smithi029.stdout:-- Logs begin at Wed 2024-01-24 13:13:42 UTC. -- 2024-01-24T13:33:26.266 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.b/config 2024-01-24T13:33:27.349 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:33:26 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:33:26.963+0000 7f8598ceb640 -1 osd.6 0 log_to_monitors true 2024-01-24T13:33:27.562 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-01-24T13:33:27.562 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-01-24T13:33:27.562 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10+0 records in 2024-01-24T13:33:27.562 INFO:teuthology.orchestra.run.smithi163.stderr:10+0 records out 2024-01-24T13:33:27.562 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114044 s, 919 MB/s 2024-01-24T13:33:27.562 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping successful for: 2024-01-24T13:33:27.891 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:27 smithi163 ceph-mon[44969]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-01-24T13:33:27.891 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:27 smithi163 ceph-mon[44969]: from='osd.6 [v2:172.21.15.29:6818/938400701,v1:172.21.15.29:6819/938400701]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-01-24T13:33:27.932 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph orch daemon add osd smithi163:vg_nvme/lv_2 2024-01-24T13:33:27.988 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:27 smithi029 ceph-mon[40651]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-01-24T13:33:27.988 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:27 smithi029 ceph-mon[40651]: from='osd.6 [v2:172.21.15.29:6818/938400701,v1:172.21.15.29:6819/938400701]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-01-24T13:33:28.054 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:27 smithi187 ceph-mon[44911]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-01-24T13:33:28.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:27 smithi187 ceph-mon[44911]: from='osd.6 [v2:172.21.15.29:6818/938400701,v1:172.21.15.29:6819/938400701]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-01-24T13:33:28.150 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.b/config 2024-01-24T13:33:28.980 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:28 smithi163 ceph-mon[44969]: from='osd.6 [v2:172.21.15.29:6818/938400701,v1:172.21.15.29:6819/938400701]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-01-24T13:33:28.981 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:28 smithi163 ceph-mon[44969]: osdmap e43: 7 total, 6 up, 7 in 2024-01-24T13:33:28.981 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:28 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-01-24T13:33:28.981 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:28 smithi163 ceph-mon[44969]: from='osd.6 [v2:172.21.15.29:6818/938400701,v1:172.21.15.29:6819/938400701]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-01-24T13:33:28.981 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:28 smithi163 ceph-mon[44969]: Detected new or changed devices on smithi029 2024-01-24T13:33:28.981 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:28 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:28.981 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:28 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:28.981 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:28 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:33:28.981 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:28 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:33:28.981 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:28 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:33:28.981 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:28 smithi163 ceph-mon[44969]: Adjusting osd_memory_target on smithi029 to 5706M 2024-01-24T13:33:28.981 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:28 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:28.981 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:28 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:33:28.981 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:28 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:33:28.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:28 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:29.099 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:33:28 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:33:28.813+0000 7f858e350700 -1 osd.6 0 waiting for initial osdmap 2024-01-24T13:33:29.099 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:33:28 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:33:28.826+0000 7f8588141700 -1 osd.6 44 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-24T13:33:29.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:28 smithi029 ceph-mon[40651]: from='osd.6 [v2:172.21.15.29:6818/938400701,v1:172.21.15.29:6819/938400701]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-01-24T13:33:29.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:28 smithi029 ceph-mon[40651]: osdmap e43: 7 total, 6 up, 7 in 2024-01-24T13:33:29.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:28 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-01-24T13:33:29.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:28 smithi029 ceph-mon[40651]: from='osd.6 [v2:172.21.15.29:6818/938400701,v1:172.21.15.29:6819/938400701]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-01-24T13:33:29.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:28 smithi029 ceph-mon[40651]: Detected new or changed devices on smithi029 2024-01-24T13:33:29.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:28 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:29.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:28 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:29.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:28 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:33:29.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:28 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:33:29.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:28 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:33:29.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:28 smithi029 ceph-mon[40651]: Adjusting osd_memory_target on smithi029 to 5706M 2024-01-24T13:33:29.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:28 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:29.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:28 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:33:29.101 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:28 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:33:29.101 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:28 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:29.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.166+0000 7fb6454a5700 1 -- 172.21.15.163:0/728856143 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb640100f00 msgr2=0x7fb640101360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:33:29.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.166+0000 7fb6454a5700 1 --2- 172.21.15.163:0/728856143 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb640100f00 0x7fb640101360 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb6300099c0 tx=0x7fb630009cd0 comp rx=0 tx=0).stop 2024-01-24T13:33:29.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.166+0000 7fb6454a5700 1 -- 172.21.15.163:0/728856143 shutdown_connections 2024-01-24T13:33:29.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.166+0000 7fb6454a5700 1 --2- 172.21.15.163:0/728856143 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb6401018a0 0x7fb640108150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:33:29.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.166+0000 7fb6454a5700 1 --2- 172.21.15.163:0/728856143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6400ffd00 0x7fb640100120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:33:29.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.166+0000 7fb6454a5700 1 --2- 172.21.15.163:0/728856143 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb640100f00 0x7fb640101360 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:33:29.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.166+0000 7fb6454a5700 1 -- 172.21.15.163:0/728856143 >> 172.21.15.163:0/728856143 conn(0x7fb6400fb2c0 msgr2=0x7fb6400fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:33:29.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.166+0000 7fb6454a5700 1 -- 172.21.15.163:0/728856143 shutdown_connections 2024-01-24T13:33:29.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.166+0000 7fb6454a5700 1 -- 172.21.15.163:0/728856143 wait complete. 2024-01-24T13:33:29.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.166+0000 7fb6454a5700 1 Processor -- start 2024-01-24T13:33:29.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.167+0000 7fb6454a5700 1 -- start start 2024-01-24T13:33:29.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.167+0000 7fb6454a5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb6400ffd00 0x7fb640196a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:33:29.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.167+0000 7fb6454a5700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb640100f00 0x7fb640196fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:33:29.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.167+0000 7fb6454a5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6401018a0 0x7fb64019e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:33:29.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.167+0000 7fb6454a5700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb64019e620 con 0x7fb6401018a0 2024-01-24T13:33:29.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.167+0000 7fb6454a5700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb64019e760 con 0x7fb640100f00 2024-01-24T13:33:29.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.167+0000 7fb6454a5700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb64019e8f0 con 0x7fb6400ffd00 2024-01-24T13:33:29.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.167+0000 7fb63e7fc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb640100f00 0x7fb640196fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:33:29.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.167+0000 7fb63e7fc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb640100f00 0x7fb640196fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:54776/0 (socket says 172.21.15.163:54776) 2024-01-24T13:33:29.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.167+0000 7fb63e7fc700 1 -- 172.21.15.163:0/1827562000 learned_addr learned my addr 172.21.15.163:0/1827562000 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-01-24T13:33:29.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.167+0000 7fb63effd700 1 --2- 172.21.15.163:0/1827562000 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb6400ffd00 0x7fb640196a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:33:29.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.167+0000 7fb63f7fe700 1 --2- 172.21.15.163:0/1827562000 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6401018a0 0x7fb64019e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:33:29.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.167+0000 7fb63e7fc700 1 -- 172.21.15.163:0/1827562000 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb6400ffd00 msgr2=0x7fb640196a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:33:29.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.167+0000 7fb63e7fc700 1 --2- 172.21.15.163:0/1827562000 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb6400ffd00 0x7fb640196a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:33:29.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.167+0000 7fb63e7fc700 1 -- 172.21.15.163:0/1827562000 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6401018a0 msgr2=0x7fb64019e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:33:29.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.167+0000 7fb63e7fc700 1 --2- 172.21.15.163:0/1827562000 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6401018a0 0x7fb64019e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:33:29.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.167+0000 7fb63e7fc700 1 -- 172.21.15.163:0/1827562000 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6300095f0 con 0x7fb640100f00 2024-01-24T13:33:29.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.167+0000 7fb63e7fc700 1 --2- 172.21.15.163:0/1827562000 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb640100f00 0x7fb640196fd0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb630009990 tx=0x7fb630004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:33:29.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.167+0000 7fb627fff700 1 -- 172.21.15.163:0/1827562000 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63001d070 con 0x7fb640100f00 2024-01-24T13:33:29.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.167+0000 7fb627fff700 1 -- 172.21.15.163:0/1827562000 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb630004090 con 0x7fb640100f00 2024-01-24T13:33:29.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.167+0000 7fb6454a5700 1 -- 172.21.15.163:0/1827562000 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb64019eb70 con 0x7fb640100f00 2024-01-24T13:33:29.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.167+0000 7fb627fff700 1 -- 172.21.15.163:0/1827562000 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb630004a00 con 0x7fb640100f00 2024-01-24T13:33:29.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.168+0000 7fb6454a5700 1 -- 172.21.15.163:0/1827562000 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb64019f060 con 0x7fb640100f00 2024-01-24T13:33:29.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.169+0000 7fb6454a5700 1 -- 172.21.15.163:0/1827562000 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb64010b500 con 0x7fb640100f00 2024-01-24T13:33:29.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.170+0000 7fb627fff700 1 -- 172.21.15.163:0/1827562000 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7fb630003bd0 con 0x7fb640100f00 2024-01-24T13:33:29.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.170+0000 7fb627fff700 1 --2- 172.21.15.163:0/1827562000 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb6280a05a0 0x7fb6280a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:33:29.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.170+0000 7fb627fff700 1 -- 172.21.15.163:0/1827562000 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4685+0+0 (secure 0 0 0) 0x7fb6300cb9d0 con 0x7fb640100f00 2024-01-24T13:33:29.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.171+0000 7fb63effd700 1 --2- 172.21.15.163:0/1827562000 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb6280a05a0 0x7fb6280a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:33:29.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.171+0000 7fb63effd700 1 --2- 172.21.15.163:0/1827562000 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb6280a05a0 0x7fb6280a2a60 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb640100d60 tx=0x7fb63400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:33:29.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.175+0000 7fb627fff700 1 -- 172.21.15.163:0/1827562000 <== mon.2 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fb630096550 con 0x7fb640100f00 2024-01-24T13:33:29.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:28 smithi187 ceph-mon[44911]: from='osd.6 [v2:172.21.15.29:6818/938400701,v1:172.21.15.29:6819/938400701]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-01-24T13:33:29.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:28 smithi187 ceph-mon[44911]: osdmap e43: 7 total, 6 up, 7 in 2024-01-24T13:33:29.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:28 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-01-24T13:33:29.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:28 smithi187 ceph-mon[44911]: from='osd.6 [v2:172.21.15.29:6818/938400701,v1:172.21.15.29:6819/938400701]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-01-24T13:33:29.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:28 smithi187 ceph-mon[44911]: Detected new or changed devices on smithi029 2024-01-24T13:33:29.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:28 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:29.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:28 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:29.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:28 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:33:29.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:28 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:33:29.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:28 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:33:29.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:28 smithi187 ceph-mon[44911]: Adjusting osd_memory_target on smithi029 to 5706M 2024-01-24T13:33:29.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:28 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:29.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:28 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:33:29.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:28 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:33:29.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:28 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:29.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:29.371+0000 7fb6454a5700 1 -- 172.21.15.163:0/1827562000 --> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fb6400698f0 con 0x7fb6280a05a0 2024-01-24T13:33:30.070 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:29 smithi163 ceph-mon[44969]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-01-24T13:33:30.070 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:29 smithi163 ceph-mon[44969]: from='osd.6 [v2:172.21.15.29:6818/938400701,v1:172.21.15.29:6819/938400701]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-01-24T13:33:30.070 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:29 smithi163 ceph-mon[44969]: osdmap e44: 7 total, 6 up, 7 in 2024-01-24T13:33:30.070 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:29 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-01-24T13:33:30.070 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:29 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-01-24T13:33:30.071 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:29 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T13:33:30.071 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:29 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T13:33:30.071 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:29 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:33:30.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:29 smithi029 ceph-mon[40651]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-01-24T13:33:30.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:29 smithi029 ceph-mon[40651]: from='osd.6 [v2:172.21.15.29:6818/938400701,v1:172.21.15.29:6819/938400701]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-01-24T13:33:30.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:29 smithi029 ceph-mon[40651]: osdmap e44: 7 total, 6 up, 7 in 2024-01-24T13:33:30.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:29 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-01-24T13:33:30.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:29 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-01-24T13:33:30.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:29 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T13:33:30.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:29 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T13:33:30.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:29 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:33:30.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:29 smithi187 ceph-mon[44911]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-01-24T13:33:30.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:29 smithi187 ceph-mon[44911]: from='osd.6 [v2:172.21.15.29:6818/938400701,v1:172.21.15.29:6819/938400701]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-01-24T13:33:30.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:29 smithi187 ceph-mon[44911]: osdmap e44: 7 total, 6 up, 7 in 2024-01-24T13:33:30.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:29 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-01-24T13:33:30.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:29 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-01-24T13:33:30.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:29 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T13:33:30.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:29 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T13:33:30.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:29 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:33:31.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:30 smithi029 ceph-mon[40651]: purged_snaps scrub starts 2024-01-24T13:33:31.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:30 smithi029 ceph-mon[40651]: purged_snaps scrub ok 2024-01-24T13:33:31.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:30 smithi029 ceph-mon[40651]: from='client.24305 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:33:31.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:30 smithi029 ceph-mon[40651]: osd.6 [v2:172.21.15.29:6818/938400701,v1:172.21.15.29:6819/938400701] boot 2024-01-24T13:33:31.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:30 smithi029 ceph-mon[40651]: osdmap e45: 7 total, 7 up, 7 in 2024-01-24T13:33:31.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:30 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-01-24T13:33:31.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:30 smithi163 ceph-mon[44969]: purged_snaps scrub starts 2024-01-24T13:33:31.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:30 smithi163 ceph-mon[44969]: purged_snaps scrub ok 2024-01-24T13:33:31.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:30 smithi163 ceph-mon[44969]: from='client.24305 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:33:31.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:30 smithi163 ceph-mon[44969]: osd.6 [v2:172.21.15.29:6818/938400701,v1:172.21.15.29:6819/938400701] boot 2024-01-24T13:33:31.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:30 smithi163 ceph-mon[44969]: osdmap e45: 7 total, 7 up, 7 in 2024-01-24T13:33:31.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:30 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-01-24T13:33:31.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:30 smithi187 ceph-mon[44911]: purged_snaps scrub starts 2024-01-24T13:33:31.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:30 smithi187 ceph-mon[44911]: purged_snaps scrub ok 2024-01-24T13:33:31.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:30 smithi187 ceph-mon[44911]: from='client.24305 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:33:31.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:30 smithi187 ceph-mon[44911]: osd.6 [v2:172.21.15.29:6818/938400701,v1:172.21.15.29:6819/938400701] boot 2024-01-24T13:33:31.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:30 smithi187 ceph-mon[44911]: osdmap e45: 7 total, 7 up, 7 in 2024-01-24T13:33:31.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:30 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-01-24T13:33:32.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:31 smithi029 ceph-mon[40651]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T13:33:32.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:31 smithi029 ceph-mon[40651]: osdmap e46: 7 total, 7 up, 7 in 2024-01-24T13:33:32.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:31 smithi029 ceph-mon[40651]: from='client.? 172.21.15.163:0/171274992' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "add3d09b-76f3-4a93-83a2-30ac94c3fad1"}]: dispatch 2024-01-24T13:33:32.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:31 smithi029 ceph-mon[40651]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "add3d09b-76f3-4a93-83a2-30ac94c3fad1"}]: dispatch 2024-01-24T13:33:32.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:31 smithi029 ceph-mon[40651]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "add3d09b-76f3-4a93-83a2-30ac94c3fad1"}]': finished 2024-01-24T13:33:32.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:31 smithi029 ceph-mon[40651]: osdmap e47: 8 total, 7 up, 8 in 2024-01-24T13:33:32.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:31 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-01-24T13:33:32.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:31 smithi029 ceph-mon[40651]: from='client.? 172.21.15.163:0/4284859492' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T13:33:32.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:31 smithi163 ceph-mon[44969]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T13:33:32.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:31 smithi163 ceph-mon[44969]: osdmap e46: 7 total, 7 up, 7 in 2024-01-24T13:33:32.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:31 smithi163 ceph-mon[44969]: from='client.? 172.21.15.163:0/171274992' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "add3d09b-76f3-4a93-83a2-30ac94c3fad1"}]: dispatch 2024-01-24T13:33:32.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:31 smithi163 ceph-mon[44969]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "add3d09b-76f3-4a93-83a2-30ac94c3fad1"}]: dispatch 2024-01-24T13:33:32.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:31 smithi163 ceph-mon[44969]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "add3d09b-76f3-4a93-83a2-30ac94c3fad1"}]': finished 2024-01-24T13:33:32.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:31 smithi163 ceph-mon[44969]: osdmap e47: 8 total, 7 up, 8 in 2024-01-24T13:33:32.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:31 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-01-24T13:33:32.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:31 smithi163 ceph-mon[44969]: from='client.? 172.21.15.163:0/4284859492' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T13:33:32.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:31 smithi187 ceph-mon[44911]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T13:33:32.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:31 smithi187 ceph-mon[44911]: osdmap e46: 7 total, 7 up, 7 in 2024-01-24T13:33:32.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:31 smithi187 ceph-mon[44911]: from='client.? 172.21.15.163:0/171274992' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "add3d09b-76f3-4a93-83a2-30ac94c3fad1"}]: dispatch 2024-01-24T13:33:32.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:31 smithi187 ceph-mon[44911]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "add3d09b-76f3-4a93-83a2-30ac94c3fad1"}]: dispatch 2024-01-24T13:33:32.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:31 smithi187 ceph-mon[44911]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "add3d09b-76f3-4a93-83a2-30ac94c3fad1"}]': finished 2024-01-24T13:33:32.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:31 smithi187 ceph-mon[44911]: osdmap e47: 8 total, 7 up, 8 in 2024-01-24T13:33:32.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:31 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-01-24T13:33:32.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:31 smithi187 ceph-mon[44911]: from='client.? 172.21.15.163:0/4284859492' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T13:33:34.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:33 smithi029 ceph-mon[40651]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T13:33:34.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:33 smithi163 ceph-mon[44969]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T13:33:34.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:33 smithi187 ceph-mon[44911]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T13:33:36.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:35 smithi029 ceph-mon[40651]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T13:33:36.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:35 smithi163 ceph-mon[44969]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T13:33:36.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:35 smithi187 ceph-mon[44911]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T13:33:37.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:37 smithi163 ceph-mon[44969]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T13:33:37.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:37 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-01-24T13:33:37.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:37 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:33:37.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:37 smithi163 ceph-mon[44969]: Deploying daemon osd.7 on smithi163 2024-01-24T13:33:38.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:37 smithi029 ceph-mon[40651]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T13:33:38.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:37 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-01-24T13:33:38.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:37 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:33:38.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:37 smithi029 ceph-mon[40651]: Deploying daemon osd.7 on smithi163 2024-01-24T13:33:38.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:37 smithi187 ceph-mon[44911]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T13:33:38.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:37 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-01-24T13:33:38.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:37 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:33:38.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:37 smithi187 ceph-mon[44911]: Deploying daemon osd.7 on smithi163 2024-01-24T13:33:40.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:39 smithi029 ceph-mon[40651]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T13:33:40.103 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:39 smithi163 ceph-mon[44969]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T13:33:40.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:39 smithi187 ceph-mon[44911]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T13:33:41.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:41 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:41.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:41 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:33:41.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:41 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:41.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:41 smithi163 ceph-mon[44969]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T13:33:42.054 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:41 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:42.054 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:41 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:33:42.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:41 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:42.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:41 smithi187 ceph-mon[44911]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T13:33:42.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:41 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:42.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:41 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:33:42.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:41 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:42.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:41 smithi029 ceph-mon[40651]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T13:33:42.876 INFO:teuthology.orchestra.run.smithi163.stdout:Created osd(s) 7 on host 'smithi163' 2024-01-24T13:33:42.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:42.876+0000 7fb627fff700 1 -- 172.21.15.163:0/1827562000 <== mgr.14152 v2:172.21.15.29:6800/2498991338 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb6400698f0 con 0x7fb6280a05a0 2024-01-24T13:33:42.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:42.878+0000 7fb6454a5700 1 -- 172.21.15.163:0/1827562000 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb6280a05a0 msgr2=0x7fb6280a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:33:42.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:42.878+0000 7fb6454a5700 1 --2- 172.21.15.163:0/1827562000 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb6280a05a0 0x7fb6280a2a60 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb640100d60 tx=0x7fb63400b410 comp rx=0 tx=0).stop 2024-01-24T13:33:42.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:42.879+0000 7fb6454a5700 1 -- 172.21.15.163:0/1827562000 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb640100f00 msgr2=0x7fb640196fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:33:42.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:42.879+0000 7fb6454a5700 1 --2- 172.21.15.163:0/1827562000 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb640100f00 0x7fb640196fd0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb630009990 tx=0x7fb630004730 comp rx=0 tx=0).stop 2024-01-24T13:33:42.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:42.879+0000 7fb6454a5700 1 -- 172.21.15.163:0/1827562000 shutdown_connections 2024-01-24T13:33:42.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:42.879+0000 7fb6454a5700 1 --2- 172.21.15.163:0/1827562000 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb6280a05a0 0x7fb6280a2a60 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:33:42.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:42.879+0000 7fb6454a5700 1 --2- 172.21.15.163:0/1827562000 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6401018a0 0x7fb64019e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:33:42.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:42.879+0000 7fb6454a5700 1 --2- 172.21.15.163:0/1827562000 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb6400ffd00 0x7fb640196a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:33:42.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:42.879+0000 7fb6454a5700 1 --2- 172.21.15.163:0/1827562000 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb640100f00 0x7fb640196fd0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:33:42.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:42.879+0000 7fb6454a5700 1 -- 172.21.15.163:0/1827562000 >> 172.21.15.163:0/1827562000 conn(0x7fb6400fb2c0 msgr2=0x7fb6400691e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:33:42.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:42.879+0000 7fb6454a5700 1 -- 172.21.15.163:0/1827562000 shutdown_connections 2024-01-24T13:33:42.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:33:42.879+0000 7fb6454a5700 1 -- 172.21.15.163:0/1827562000 wait complete. 2024-01-24T13:33:43.308 DEBUG:teuthology.orchestra.run.smithi163:osd.7> sudo journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@osd.7.service 2024-01-24T13:33:43.310 INFO:tasks.cephadm:Deploying osd.8 on smithi187 with /dev/vg_nvme/lv_2... 2024-01-24T13:33:43.310 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-01-24T13:33:43.334 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:43 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:43.334 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:43 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:43.334 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:43 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:33:43.334 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:43 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:33:43.335 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:43 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:43.335 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:43 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:43.335 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:43 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:43.335 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:43 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:33:43.340 INFO:journalctl@ceph.osd.7.smithi163.stdout:-- Logs begin at Wed 2024-01-24 13:14:09 UTC. -- 2024-01-24T13:33:43.343 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:43 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:43.344 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:43 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:43.344 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:43 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:33:43.344 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:43 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:33:43.344 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:43 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:43.344 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:43 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:43.344 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:43 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:43.344 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:43 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:33:43.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:43 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:43.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:43 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:43.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:43 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:33:43.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:43 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:33:43.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:43 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:43.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:43 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:43.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:43 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:43.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:43 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:33:43.544 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.c/config 2024-01-24T13:33:44.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:44 smithi187 ceph-mon[44911]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T13:33:44.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:44 smithi029 ceph-mon[40651]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T13:33:44.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:44 smithi163 ceph-mon[44969]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T13:33:44.838 INFO:teuthology.orchestra.run.smithi187.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-01-24T13:33:44.838 INFO:teuthology.orchestra.run.smithi187.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-01-24T13:33:44.838 INFO:teuthology.orchestra.run.smithi187.stderr: stderr: 10+0 records in 2024-01-24T13:33:44.838 INFO:teuthology.orchestra.run.smithi187.stderr:10+0 records out 2024-01-24T13:33:44.839 INFO:teuthology.orchestra.run.smithi187.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111884 s, 937 MB/s 2024-01-24T13:33:44.839 INFO:teuthology.orchestra.run.smithi187.stderr:--> Zapping successful for: 2024-01-24T13:33:44.982 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:33:44 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:33:44.700+0000 7f815e826640 -1 osd.7 0 log_to_monitors true 2024-01-24T13:33:45.251 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:45 smithi187 ceph-mon[44911]: from='osd.7 [v2:172.21.15.163:6816/2783976092,v1:172.21.15.163:6817/2783976092]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-01-24T13:33:45.252 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:45 smithi187 ceph-mon[44911]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-01-24T13:33:45.253 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph orch daemon add osd smithi187:vg_nvme/lv_2 2024-01-24T13:33:45.338 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:45 smithi163 ceph-mon[44969]: from='osd.7 [v2:172.21.15.163:6816/2783976092,v1:172.21.15.163:6817/2783976092]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-01-24T13:33:45.338 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:45 smithi163 ceph-mon[44969]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-01-24T13:33:45.512 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.c/config 2024-01-24T13:33:45.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:45 smithi029 ceph-mon[40651]: from='osd.7 [v2:172.21.15.163:6816/2783976092,v1:172.21.15.163:6817/2783976092]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-01-24T13:33:45.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:45 smithi029 ceph-mon[40651]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-01-24T13:33:46.260 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:46 smithi187 ceph-mon[44911]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T13:33:46.260 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:46 smithi187 ceph-mon[44911]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-01-24T13:33:46.260 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:46 smithi187 ceph-mon[44911]: osdmap e48: 8 total, 7 up, 8 in 2024-01-24T13:33:46.260 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:46 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-01-24T13:33:46.260 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:46 smithi187 ceph-mon[44911]: from='osd.7 [v2:172.21.15.163:6816/2783976092,v1:172.21.15.163:6817/2783976092]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-01-24T13:33:46.260 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:46 smithi187 ceph-mon[44911]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-01-24T13:33:46.260 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:46 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:46.260 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:46 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:46.260 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:46 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:33:46.261 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:46 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:33:46.261 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:46 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:33:46.261 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:46 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:46.261 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:46 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:33:46.261 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:46 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:33:46.261 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:46 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:46.419 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.418+0000 7fb1882b9700 1 -- 172.21.15.187:0/906935710 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb180101870 msgr2=0x7fb180108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:33:46.419 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.418+0000 7fb1882b9700 1 --2- 172.21.15.187:0/906935710 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb180101870 0x7fb180108120 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb17800b0a0 tx=0x7fb17800b3b0 comp rx=0 tx=0).stop 2024-01-24T13:33:46.419 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.418+0000 7fb1882b9700 1 -- 172.21.15.187:0/906935710 shutdown_connections 2024-01-24T13:33:46.419 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.418+0000 7fb1882b9700 1 --2- 172.21.15.187:0/906935710 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb180101870 0x7fb180108120 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:33:46.419 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.418+0000 7fb1882b9700 1 --2- 172.21.15.187:0/906935710 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1800ffcd0 0x7fb1801000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:33:46.419 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.418+0000 7fb1882b9700 1 --2- 172.21.15.187:0/906935710 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb180100ed0 0x7fb180101330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:33:46.419 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.418+0000 7fb1882b9700 1 -- 172.21.15.187:0/906935710 >> 172.21.15.187:0/906935710 conn(0x7fb1800fb250 msgr2=0x7fb1800fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:33:46.419 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.418+0000 7fb1882b9700 1 -- 172.21.15.187:0/906935710 shutdown_connections 2024-01-24T13:33:46.419 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.418+0000 7fb1882b9700 1 -- 172.21.15.187:0/906935710 wait complete. 2024-01-24T13:33:46.422 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.419+0000 7fb1882b9700 1 Processor -- start 2024-01-24T13:33:46.422 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.419+0000 7fb1882b9700 1 -- start start 2024-01-24T13:33:46.422 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.419+0000 7fb1882b9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1800ffcd0 0x7fb180195a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:33:46.422 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.419+0000 7fb1882b9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb180100ed0 0x7fb180195fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:33:46.422 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.419+0000 7fb1882b9700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb180101870 0x7fb18019d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:33:46.422 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.419+0000 7fb1882b9700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb18019d600 con 0x7fb180100ed0 2024-01-24T13:33:46.422 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.419+0000 7fb1882b9700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb18019d740 con 0x7fb180101870 2024-01-24T13:33:46.422 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.419+0000 7fb1882b9700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb18019d8d0 con 0x7fb1800ffcd0 2024-01-24T13:33:46.422 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.419+0000 7fb186055700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1800ffcd0 0x7fb180195a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:33:46.422 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.419+0000 7fb186055700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1800ffcd0 0x7fb180195a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:47046/0 (socket says 172.21.15.187:47046) 2024-01-24T13:33:46.422 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.419+0000 7fb186055700 1 -- 172.21.15.187:0/3835957085 learned_addr learned my addr 172.21.15.187:0/3835957085 (peer_addr_for_me v2:172.21.15.187:0/0) 2024-01-24T13:33:46.423 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.419+0000 7fb186856700 1 --2- 172.21.15.187:0/3835957085 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb180101870 0x7fb18019d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:33:46.423 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.419+0000 7fb185854700 1 --2- 172.21.15.187:0/3835957085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb180100ed0 0x7fb180195fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:33:46.423 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.419+0000 7fb186055700 1 -- 172.21.15.187:0/3835957085 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb180101870 msgr2=0x7fb18019d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:33:46.423 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.419+0000 7fb186055700 1 --2- 172.21.15.187:0/3835957085 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb180101870 0x7fb18019d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:33:46.423 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.419+0000 7fb186055700 1 -- 172.21.15.187:0/3835957085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb180100ed0 msgr2=0x7fb180195fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:33:46.423 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.419+0000 7fb186055700 1 --2- 172.21.15.187:0/3835957085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb180100ed0 0x7fb180195fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:33:46.423 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.419+0000 7fb186055700 1 -- 172.21.15.187:0/3835957085 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb178009d00 con 0x7fb1800ffcd0 2024-01-24T13:33:46.423 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.419+0000 7fb185854700 1 --2- 172.21.15.187:0/3835957085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb180100ed0 0x7fb180195fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T13:33:46.423 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.419+0000 7fb186055700 1 --2- 172.21.15.187:0/3835957085 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1800ffcd0 0x7fb180195a70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb17c00d9b0 tx=0x7fb17c00dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:33:46.423 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.420+0000 7fb1777fe700 1 -- 172.21.15.187:0/3835957085 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17c00b950 con 0x7fb1800ffcd0 2024-01-24T13:33:46.423 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.420+0000 7fb1882b9700 1 -- 172.21.15.187:0/3835957085 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb18019dbb0 con 0x7fb1800ffcd0 2024-01-24T13:33:46.423 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.420+0000 7fb1777fe700 1 -- 172.21.15.187:0/3835957085 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb17c007620 con 0x7fb1800ffcd0 2024-01-24T13:33:46.424 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.420+0000 7fb1777fe700 1 -- 172.21.15.187:0/3835957085 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17c00f3c0 con 0x7fb1800ffcd0 2024-01-24T13:33:46.424 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.420+0000 7fb1882b9700 1 -- 172.21.15.187:0/3835957085 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb18019e100 con 0x7fb1800ffcd0 2024-01-24T13:33:46.424 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.420+0000 7fb1882b9700 1 -- 172.21.15.187:0/3835957085 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb180107270 con 0x7fb1800ffcd0 2024-01-24T13:33:46.424 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.422+0000 7fb1777fe700 1 -- 172.21.15.187:0/3835957085 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7fb17c010640 con 0x7fb1800ffcd0 2024-01-24T13:33:46.425 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.422+0000 7fb1777fe700 1 --2- 172.21.15.187:0/3835957085 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb16c0a0520 0x7fb16c0a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:33:46.425 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.423+0000 7fb1777fe700 1 -- 172.21.15.187:0/3835957085 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5117+0+0 (secure 0 0 0) 0x7fb17c0cb3c0 con 0x7fb1800ffcd0 2024-01-24T13:33:46.425 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.423+0000 7fb185854700 1 --2- 172.21.15.187:0/3835957085 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb16c0a0520 0x7fb16c0a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:33:46.425 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.423+0000 7fb185854700 1 --2- 172.21.15.187:0/3835957085 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb16c0a0520 0x7fb16c0a29e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb170005fd0 tx=0x7fb170005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:33:46.426 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.426+0000 7fb1777fe700 1 -- 172.21.15.187:0/3835957085 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fb17c095be0 con 0x7fb1800ffcd0 2024-01-24T13:33:46.482 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:33:46 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:33:46.109+0000 7f8153e8b700 -1 osd.7 0 waiting for initial osdmap 2024-01-24T13:33:46.482 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:33:46 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:33:46.118+0000 7f814e47d700 -1 osd.7 49 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-24T13:33:46.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:46 smithi163 ceph-mon[44969]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T13:33:46.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:46 smithi163 ceph-mon[44969]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-01-24T13:33:46.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:46 smithi163 ceph-mon[44969]: osdmap e48: 8 total, 7 up, 8 in 2024-01-24T13:33:46.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:46 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-01-24T13:33:46.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:46 smithi163 ceph-mon[44969]: from='osd.7 [v2:172.21.15.163:6816/2783976092,v1:172.21.15.163:6817/2783976092]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-01-24T13:33:46.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:46 smithi163 ceph-mon[44969]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-01-24T13:33:46.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:46 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:46.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:46 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:46.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:46 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:33:46.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:46 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:33:46.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:46 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:33:46.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:46 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:46.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:46 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:33:46.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:46 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:33:46.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:46 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:46.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:46 smithi029 ceph-mon[40651]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T13:33:46.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:46 smithi029 ceph-mon[40651]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-01-24T13:33:46.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:46 smithi029 ceph-mon[40651]: osdmap e48: 8 total, 7 up, 8 in 2024-01-24T13:33:46.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:46 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-01-24T13:33:46.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:46 smithi029 ceph-mon[40651]: from='osd.7 [v2:172.21.15.163:6816/2783976092,v1:172.21.15.163:6817/2783976092]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-01-24T13:33:46.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:46 smithi029 ceph-mon[40651]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-01-24T13:33:46.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:46 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:46.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:46 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:46.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:46 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:33:46.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:46 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:33:46.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:46 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:33:46.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:46 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:46.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:46 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:33:46.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:46 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:33:46.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:46 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:46.620 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:46.619+0000 7fb1882b9700 1 -- 172.21.15.187:0/3835957085 --> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fb180061180 con 0x7fb16c0a0520 2024-01-24T13:33:47.264 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:47 smithi187 ceph-mon[44911]: Detected new or changed devices on smithi163 2024-01-24T13:33:47.264 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:47 smithi187 ceph-mon[44911]: Adjusting osd_memory_target on smithi163 to 5704M 2024-01-24T13:33:47.264 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:47 smithi187 ceph-mon[44911]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-01-24T13:33:47.265 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:47 smithi187 ceph-mon[44911]: osdmap e49: 8 total, 7 up, 8 in 2024-01-24T13:33:47.265 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:47 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-01-24T13:33:47.265 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:47 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-01-24T13:33:47.265 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:47 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T13:33:47.265 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:47 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T13:33:47.265 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:47 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:33:47.265 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:47 smithi187 ceph-mon[44911]: osd.7 [v2:172.21.15.163:6816/2783976092,v1:172.21.15.163:6817/2783976092] boot 2024-01-24T13:33:47.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:47 smithi163 ceph-mon[44969]: Detected new or changed devices on smithi163 2024-01-24T13:33:47.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:47 smithi163 ceph-mon[44969]: Adjusting osd_memory_target on smithi163 to 5704M 2024-01-24T13:33:47.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:47 smithi163 ceph-mon[44969]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-01-24T13:33:47.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:47 smithi163 ceph-mon[44969]: osdmap e49: 8 total, 7 up, 8 in 2024-01-24T13:33:47.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:47 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-01-24T13:33:47.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:47 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-01-24T13:33:47.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:47 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T13:33:47.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:47 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T13:33:47.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:47 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:33:47.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:47 smithi163 ceph-mon[44969]: osd.7 [v2:172.21.15.163:6816/2783976092,v1:172.21.15.163:6817/2783976092] boot 2024-01-24T13:33:47.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:47 smithi029 ceph-mon[40651]: Detected new or changed devices on smithi163 2024-01-24T13:33:47.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:47 smithi029 ceph-mon[40651]: Adjusting osd_memory_target on smithi163 to 5704M 2024-01-24T13:33:47.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:47 smithi029 ceph-mon[40651]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-01-24T13:33:47.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:47 smithi029 ceph-mon[40651]: osdmap e49: 8 total, 7 up, 8 in 2024-01-24T13:33:47.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:47 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-01-24T13:33:47.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:47 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-01-24T13:33:47.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:47 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T13:33:47.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:47 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T13:33:47.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:47 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:33:47.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:47 smithi029 ceph-mon[40651]: osd.7 [v2:172.21.15.163:6816/2783976092,v1:172.21.15.163:6817/2783976092] boot 2024-01-24T13:33:48.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:48 smithi163 ceph-mon[44969]: purged_snaps scrub starts 2024-01-24T13:33:48.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:48 smithi163 ceph-mon[44969]: purged_snaps scrub ok 2024-01-24T13:33:48.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:48 smithi163 ceph-mon[44969]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:33:48.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:48 smithi163 ceph-mon[44969]: pgmap v137: 1 pgs: 1 unknown; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T13:33:48.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:48 smithi163 ceph-mon[44969]: osdmap e50: 8 total, 8 up, 8 in 2024-01-24T13:33:48.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:48 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-01-24T13:33:48.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:48 smithi163 ceph-mon[44969]: osdmap e51: 8 total, 8 up, 8 in 2024-01-24T13:33:48.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:48 smithi187 ceph-mon[44911]: purged_snaps scrub starts 2024-01-24T13:33:48.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:48 smithi187 ceph-mon[44911]: purged_snaps scrub ok 2024-01-24T13:33:48.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:48 smithi187 ceph-mon[44911]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:33:48.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:48 smithi187 ceph-mon[44911]: pgmap v137: 1 pgs: 1 unknown; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T13:33:48.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:48 smithi187 ceph-mon[44911]: osdmap e50: 8 total, 8 up, 8 in 2024-01-24T13:33:48.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:48 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-01-24T13:33:48.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:48 smithi187 ceph-mon[44911]: osdmap e51: 8 total, 8 up, 8 in 2024-01-24T13:33:48.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:48 smithi029 ceph-mon[40651]: purged_snaps scrub starts 2024-01-24T13:33:48.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:48 smithi029 ceph-mon[40651]: purged_snaps scrub ok 2024-01-24T13:33:48.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:48 smithi029 ceph-mon[40651]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:33:48.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:48 smithi029 ceph-mon[40651]: pgmap v137: 1 pgs: 1 unknown; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T13:33:48.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:48 smithi029 ceph-mon[40651]: osdmap e50: 8 total, 8 up, 8 in 2024-01-24T13:33:48.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:48 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-01-24T13:33:48.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:48 smithi029 ceph-mon[40651]: osdmap e51: 8 total, 8 up, 8 in 2024-01-24T13:33:49.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:49 smithi163 ceph-mon[44969]: from='client.? 172.21.15.187:0/3852053796' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71044d6a-b55f-4bcf-94c5-79f71eb7b422"}]: dispatch 2024-01-24T13:33:49.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:49 smithi163 ceph-mon[44969]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71044d6a-b55f-4bcf-94c5-79f71eb7b422"}]: dispatch 2024-01-24T13:33:49.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:49 smithi163 ceph-mon[44969]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71044d6a-b55f-4bcf-94c5-79f71eb7b422"}]': finished 2024-01-24T13:33:49.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:49 smithi163 ceph-mon[44969]: osdmap e52: 9 total, 8 up, 9 in 2024-01-24T13:33:49.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:49 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T13:33:49.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:49 smithi163 ceph-mon[44969]: from='client.? 172.21.15.187:0/1273382637' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T13:33:49.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:49 smithi187 ceph-mon[44911]: from='client.? 172.21.15.187:0/3852053796' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71044d6a-b55f-4bcf-94c5-79f71eb7b422"}]: dispatch 2024-01-24T13:33:49.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:49 smithi187 ceph-mon[44911]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71044d6a-b55f-4bcf-94c5-79f71eb7b422"}]: dispatch 2024-01-24T13:33:49.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:49 smithi187 ceph-mon[44911]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71044d6a-b55f-4bcf-94c5-79f71eb7b422"}]': finished 2024-01-24T13:33:49.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:49 smithi187 ceph-mon[44911]: osdmap e52: 9 total, 8 up, 9 in 2024-01-24T13:33:49.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:49 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T13:33:49.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:49 smithi187 ceph-mon[44911]: from='client.? 172.21.15.187:0/1273382637' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T13:33:49.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:49 smithi029 ceph-mon[40651]: from='client.? 172.21.15.187:0/3852053796' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71044d6a-b55f-4bcf-94c5-79f71eb7b422"}]: dispatch 2024-01-24T13:33:49.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:49 smithi029 ceph-mon[40651]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71044d6a-b55f-4bcf-94c5-79f71eb7b422"}]: dispatch 2024-01-24T13:33:49.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:49 smithi029 ceph-mon[40651]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71044d6a-b55f-4bcf-94c5-79f71eb7b422"}]': finished 2024-01-24T13:33:49.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:49 smithi029 ceph-mon[40651]: osdmap e52: 9 total, 8 up, 9 in 2024-01-24T13:33:49.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:49 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T13:33:49.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:49 smithi029 ceph-mon[40651]: from='client.? 172.21.15.187:0/1273382637' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T13:33:50.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:50 smithi163 ceph-mon[44969]: pgmap v141: 1 pgs: 1 unknown; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T13:33:50.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:50 smithi163 ceph-mon[44969]: osdmap e53: 9 total, 8 up, 9 in 2024-01-24T13:33:50.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:50 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T13:33:50.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:50 smithi187 ceph-mon[44911]: pgmap v141: 1 pgs: 1 unknown; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T13:33:50.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:50 smithi187 ceph-mon[44911]: osdmap e53: 9 total, 8 up, 9 in 2024-01-24T13:33:50.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:50 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T13:33:50.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:50 smithi029 ceph-mon[40651]: pgmap v141: 1 pgs: 1 unknown; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T13:33:50.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:50 smithi029 ceph-mon[40651]: osdmap e53: 9 total, 8 up, 9 in 2024-01-24T13:33:50.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:50 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T13:33:51.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:51 smithi029 ceph-mon[40651]: osdmap e54: 9 total, 8 up, 9 in 2024-01-24T13:33:51.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:51 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T13:33:51.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:51 smithi029 ceph-mon[40651]: pgmap v144: 1 pgs: 1 unknown; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T13:33:51.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:51 smithi163 ceph-mon[44969]: osdmap e54: 9 total, 8 up, 9 in 2024-01-24T13:33:51.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:51 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T13:33:51.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:51 smithi163 ceph-mon[44969]: pgmap v144: 1 pgs: 1 unknown; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T13:33:51.804 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:51 smithi187 ceph-mon[44911]: osdmap e54: 9 total, 8 up, 9 in 2024-01-24T13:33:51.804 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:51 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T13:33:51.804 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:51 smithi187 ceph-mon[44911]: pgmap v144: 1 pgs: 1 unknown; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T13:33:54.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:53 smithi029 ceph-mon[40651]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T13:33:54.134 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:53 smithi187 ceph-mon[44911]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T13:33:54.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:53 smithi163 ceph-mon[44969]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T13:33:55.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:54 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-01-24T13:33:55.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:54 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:33:55.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:54 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-01-24T13:33:55.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:54 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:33:55.247 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:54 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-01-24T13:33:55.248 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:54 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:33:56.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:55 smithi029 ceph-mon[40651]: Deploying daemon osd.8 on smithi187 2024-01-24T13:33:56.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:55 smithi029 ceph-mon[40651]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T13:33:56.113 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:55 smithi187 ceph-mon[44911]: Deploying daemon osd.8 on smithi187 2024-01-24T13:33:56.114 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:55 smithi187 ceph-mon[44911]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T13:33:56.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:55 smithi163 ceph-mon[44969]: Deploying daemon osd.8 on smithi187 2024-01-24T13:33:56.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:55 smithi163 ceph-mon[44969]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T13:33:58.054 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:57 smithi187 ceph-mon[44911]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T13:33:58.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:57 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:58.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:57 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:58.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:57 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:33:58.098 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:57 smithi029 ceph-mon[40651]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T13:33:58.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:57 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:58.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:57 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:58.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:57 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:33:58.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:57 smithi163 ceph-mon[44969]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T13:33:58.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:57 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:58.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:57 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:33:58.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:57 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:33:59.873 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:59.872+0000 7fb1777fe700 1 -- 172.21.15.187:0/3835957085 <== mgr.14152 v2:172.21.15.29:6800/2498991338 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb180061180 con 0x7fb16c0a0520 2024-01-24T13:33:59.873 INFO:teuthology.orchestra.run.smithi187.stdout:Created osd(s) 8 on host 'smithi187' 2024-01-24T13:33:59.875 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:59.874+0000 7fb1882b9700 1 -- 172.21.15.187:0/3835957085 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb16c0a0520 msgr2=0x7fb16c0a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:33:59.875 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:59.874+0000 7fb1882b9700 1 --2- 172.21.15.187:0/3835957085 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb16c0a0520 0x7fb16c0a29e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb170005fd0 tx=0x7fb170005db0 comp rx=0 tx=0).stop 2024-01-24T13:33:59.875 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:59.874+0000 7fb1882b9700 1 -- 172.21.15.187:0/3835957085 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1800ffcd0 msgr2=0x7fb180195a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:33:59.875 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:59.874+0000 7fb1882b9700 1 --2- 172.21.15.187:0/3835957085 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1800ffcd0 0x7fb180195a70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb17c00d9b0 tx=0x7fb17c00dcc0 comp rx=0 tx=0).stop 2024-01-24T13:33:59.875 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:59.874+0000 7fb1882b9700 1 -- 172.21.15.187:0/3835957085 shutdown_connections 2024-01-24T13:33:59.875 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:59.874+0000 7fb1882b9700 1 --2- 172.21.15.187:0/3835957085 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb180101870 0x7fb18019d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:33:59.875 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:59.874+0000 7fb1882b9700 1 --2- 172.21.15.187:0/3835957085 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1800ffcd0 0x7fb180195a70 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:33:59.875 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:59.874+0000 7fb1882b9700 1 --2- 172.21.15.187:0/3835957085 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb16c0a0520 0x7fb16c0a29e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:33:59.875 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:59.874+0000 7fb1882b9700 1 --2- 172.21.15.187:0/3835957085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb180100ed0 0x7fb180195fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:33:59.876 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:59.874+0000 7fb1882b9700 1 -- 172.21.15.187:0/3835957085 >> 172.21.15.187:0/3835957085 conn(0x7fb1800fb250 msgr2=0x7fb1801043b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:33:59.876 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:59.874+0000 7fb1882b9700 1 -- 172.21.15.187:0/3835957085 shutdown_connections 2024-01-24T13:33:59.876 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:33:59.874+0000 7fb1882b9700 1 -- 172.21.15.187:0/3835957085 wait complete. 2024-01-24T13:34:00.020 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:59 smithi187 ceph-mon[44911]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T13:34:00.020 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:59 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:00.020 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:59 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:00.020 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:59 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:00.021 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:59 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:34:00.021 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:33:59 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:00.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:59 smithi029 ceph-mon[40651]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T13:34:00.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:59 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:00.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:59 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:00.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:59 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:00.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:59 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:34:00.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:33:59 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:00.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:59 smithi163 ceph-mon[44969]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T13:34:00.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:59 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:00.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:59 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:00.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:59 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:00.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:59 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:34:00.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:33:59 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:00.243 DEBUG:teuthology.orchestra.run.smithi187:osd.8> sudo journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@osd.8.service 2024-01-24T13:34:00.245 INFO:tasks.cephadm:Deploying osd.9 on smithi029 with /dev/vg_nvme/lv_1... 2024-01-24T13:34:00.245 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-01-24T13:34:00.290 INFO:journalctl@ceph.osd.8.smithi187.stdout:-- Logs begin at Wed 2024-01-24 13:13:23 UTC. -- 2024-01-24T13:34:00.513 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T13:34:00.988 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:00 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:00.988 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:00 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:00.988 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:00 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:34:01.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:00 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:01.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:00 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:01.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:00 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:34:01.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:00 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:01.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:00 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:01.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:00 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:34:01.869 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-01-24T13:34:01.869 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-01-24T13:34:01.869 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10+0 records in 2024-01-24T13:34:01.869 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2024-01-24T13:34:01.869 INFO:teuthology.orchestra.run.smithi029.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0164675 s, 637 MB/s 2024-01-24T13:34:01.869 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping successful for: 2024-01-24T13:34:02.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:34:01 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:34:01.769+0000 7f1bfc459640 -1 osd.8 0 log_to_monitors true 2024-01-24T13:34:02.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:01 smithi187 ceph-mon[44911]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T13:34:02.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:01 smithi187 ceph-mon[44911]: from='osd.8 [v2:172.21.15.187:6816/2856803882,v1:172.21.15.187:6817/2856803882]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-01-24T13:34:02.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:01 smithi187 ceph-mon[44911]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-01-24T13:34:02.192 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:01 smithi029 ceph-mon[40651]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T13:34:02.192 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:01 smithi029 ceph-mon[40651]: from='osd.8 [v2:172.21.15.187:6816/2856803882,v1:172.21.15.187:6817/2856803882]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-01-24T13:34:02.192 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:01 smithi029 ceph-mon[40651]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-01-24T13:34:02.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:01 smithi163 ceph-mon[44969]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T13:34:02.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:01 smithi163 ceph-mon[44969]: from='osd.8 [v2:172.21.15.187:6816/2856803882,v1:172.21.15.187:6817/2856803882]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-01-24T13:34:02.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:01 smithi163 ceph-mon[44969]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-01-24T13:34:02.275 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph orch daemon add osd smithi029:vg_nvme/lv_1 2024-01-24T13:34:02.534 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T13:34:02.955 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:02 smithi029 ceph-mon[40651]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-01-24T13:34:02.956 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:02 smithi029 ceph-mon[40651]: osdmap e55: 9 total, 8 up, 9 in 2024-01-24T13:34:02.956 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:02 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T13:34:02.956 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:02 smithi029 ceph-mon[40651]: from='osd.8 [v2:172.21.15.187:6816/2856803882,v1:172.21.15.187:6817/2856803882]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-01-24T13:34:02.956 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:02 smithi029 ceph-mon[40651]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-01-24T13:34:02.956 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:02 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:02.956 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:02 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:02.956 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:02 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:02.956 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:02 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:02.956 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:02 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:02.956 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:02 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:02.956 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:02 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:02.956 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:02 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:34:02.956 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:02 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:03.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:02 smithi163 ceph-mon[44969]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-01-24T13:34:03.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:02 smithi163 ceph-mon[44969]: osdmap e55: 9 total, 8 up, 9 in 2024-01-24T13:34:03.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:02 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T13:34:03.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:02 smithi163 ceph-mon[44969]: from='osd.8 [v2:172.21.15.187:6816/2856803882,v1:172.21.15.187:6817/2856803882]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-01-24T13:34:03.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:02 smithi163 ceph-mon[44969]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-01-24T13:34:03.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:02 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:03.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:02 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:03.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:02 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:03.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:02 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:03.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:02 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:03.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:02 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:03.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:02 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:03.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:02 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:34:03.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:02 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:03.304 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:34:02 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:34:02.878+0000 7f1bf1abe700 -1 osd.8 0 waiting for initial osdmap 2024-01-24T13:34:03.304 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:34:02 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:34:02.890+0000 7f1beb8af700 -1 osd.8 56 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-24T13:34:03.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:02 smithi187 ceph-mon[44911]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-01-24T13:34:03.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:02 smithi187 ceph-mon[44911]: osdmap e55: 9 total, 8 up, 9 in 2024-01-24T13:34:03.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:02 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T13:34:03.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:02 smithi187 ceph-mon[44911]: from='osd.8 [v2:172.21.15.187:6816/2856803882,v1:172.21.15.187:6817/2856803882]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-01-24T13:34:03.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:02 smithi187 ceph-mon[44911]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-01-24T13:34:03.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:02 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:03.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:02 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:03.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:02 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:03.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:02 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:03.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:02 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:03.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:02 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:03.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:02 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:03.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:02 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:34:03.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:02 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:03.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.521+0000 7f36d1745700 1 -- 172.21.15.29:0/2213142919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36cc100ed0 msgr2=0x7f36cc101330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:34:03.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.521+0000 7f36d1745700 1 --2- 172.21.15.29:0/2213142919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36cc100ed0 0x7f36cc101330 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f36b40099c0 tx=0x7f36b4009cd0 comp rx=0 tx=0).stop 2024-01-24T13:34:03.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.521+0000 7f36d1745700 1 -- 172.21.15.29:0/2213142919 shutdown_connections 2024-01-24T13:34:03.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.521+0000 7f36d1745700 1 --2- 172.21.15.29:0/2213142919 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36cc101870 0x7f36cc108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:03.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.521+0000 7f36d1745700 1 --2- 172.21.15.29:0/2213142919 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f36cc0ffcd0 0x7f36cc1000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:03.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.521+0000 7f36d1745700 1 --2- 172.21.15.29:0/2213142919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36cc100ed0 0x7f36cc101330 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:03.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.522+0000 7f36d1745700 1 -- 172.21.15.29:0/2213142919 >> 172.21.15.29:0/2213142919 conn(0x7f36cc0fb250 msgr2=0x7f36cc0fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:34:03.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.522+0000 7f36d1745700 1 -- 172.21.15.29:0/2213142919 shutdown_connections 2024-01-24T13:34:03.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.522+0000 7f36d1745700 1 -- 172.21.15.29:0/2213142919 wait complete. 2024-01-24T13:34:03.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.522+0000 7f36d1745700 1 Processor -- start 2024-01-24T13:34:03.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.522+0000 7f36d1745700 1 -- start start 2024-01-24T13:34:03.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.522+0000 7f36d1745700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f36cc0ffcd0 0x7f36cc103c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:34:03.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.522+0000 7f36d1745700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36cc100ed0 0x7f36cc1022b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:34:03.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.523+0000 7f36d1745700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36cc101870 0x7f36cc1027f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:34:03.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.523+0000 7f36d1745700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36cc102f70 con 0x7f36cc100ed0 2024-01-24T13:34:03.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.523+0000 7f36d1745700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36cc1030e0 con 0x7f36cc0ffcd0 2024-01-24T13:34:03.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.523+0000 7f36d1745700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36cc1a0a40 con 0x7f36cc101870 2024-01-24T13:34:03.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.523+0000 7f36c3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36cc100ed0 0x7f36cc1022b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:34:03.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.523+0000 7f36c3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36cc100ed0 0x7f36cc1022b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55400/0 (socket says 172.21.15.29:55400) 2024-01-24T13:34:03.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.523+0000 7f36c3fff700 1 -- 172.21.15.29:0/1806075399 learned_addr learned my addr 172.21.15.29:0/1806075399 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:34:03.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.523+0000 7f36caffd700 1 --2- 172.21.15.29:0/1806075399 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f36cc0ffcd0 0x7f36cc103c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:34:03.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.523+0000 7f36cb7fe700 1 --2- 172.21.15.29:0/1806075399 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36cc101870 0x7f36cc1027f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:34:03.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.523+0000 7f36c3fff700 1 -- 172.21.15.29:0/1806075399 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36cc101870 msgr2=0x7f36cc1027f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:34:03.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.523+0000 7f36c3fff700 1 --2- 172.21.15.29:0/1806075399 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36cc101870 0x7f36cc1027f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:03.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.523+0000 7f36c3fff700 1 -- 172.21.15.29:0/1806075399 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f36cc0ffcd0 msgr2=0x7f36cc103c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:34:03.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.523+0000 7f36c3fff700 1 --2- 172.21.15.29:0/1806075399 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f36cc0ffcd0 0x7f36cc103c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:03.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.523+0000 7f36c3fff700 1 -- 172.21.15.29:0/1806075399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b40095f0 con 0x7f36cc100ed0 2024-01-24T13:34:03.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.523+0000 7f36cb7fe700 1 --2- 172.21.15.29:0/1806075399 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36cc101870 0x7f36cc1027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T13:34:03.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.523+0000 7f36c3fff700 1 --2- 172.21.15.29:0/1806075399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36cc100ed0 0x7f36cc1022b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f36b4009990 tx=0x7f36b4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:34:03.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.523+0000 7f36c8ff9700 1 -- 172.21.15.29:0/1806075399 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b401d070 con 0x7f36cc100ed0 2024-01-24T13:34:03.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.523+0000 7f36c8ff9700 1 -- 172.21.15.29:0/1806075399 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f36b4004090 con 0x7f36cc100ed0 2024-01-24T13:34:03.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.523+0000 7f36c8ff9700 1 -- 172.21.15.29:0/1806075399 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b4004a00 con 0x7f36cc100ed0 2024-01-24T13:34:03.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.523+0000 7f36d1745700 1 -- 172.21.15.29:0/1806075399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36cc1a0cc0 con 0x7f36cc100ed0 2024-01-24T13:34:03.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.524+0000 7f36d1745700 1 -- 172.21.15.29:0/1806075399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36cc1a1110 con 0x7f36cc100ed0 2024-01-24T13:34:03.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.524+0000 7f36d1745700 1 -- 172.21.15.29:0/1806075399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36cc10b4d0 con 0x7f36cc100ed0 2024-01-24T13:34:03.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.526+0000 7f36c8ff9700 1 -- 172.21.15.29:0/1806075399 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f36b4003bd0 con 0x7f36cc100ed0 2024-01-24T13:34:03.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.527+0000 7f36c8ff9700 1 --2- 172.21.15.29:0/1806075399 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f36b80a05a0 0x7f36b80a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:34:03.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.527+0000 7f36caffd700 1 --2- 172.21.15.29:0/1806075399 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f36b80a05a0 0x7f36b80a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:34:03.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.527+0000 7f36c8ff9700 1 -- 172.21.15.29:0/1806075399 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5549+0+0 (secure 0 0 0) 0x7f36b40cc110 con 0x7f36cc100ed0 2024-01-24T13:34:03.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.528+0000 7f36caffd700 1 --2- 172.21.15.29:0/1806075399 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f36b80a05a0 0x7f36b80a2a60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f36cc100d30 tx=0x7f36bc00a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:34:03.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.532+0000 7f36c8ff9700 1 -- 172.21.15.29:0/1806075399 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f36b4096810 con 0x7f36cc100ed0 2024-01-24T13:34:03.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:03.748+0000 7f36d1745700 1 -- 172.21.15.29:0/1806075399 --> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f36cc068fd0 con 0x7f36b80a05a0 2024-01-24T13:34:04.161 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:03 smithi029 ceph-mon[40651]: Detected new or changed devices on smithi187 2024-01-24T13:34:04.161 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:03 smithi029 ceph-mon[40651]: Adjusting osd_memory_target on smithi187 to 5704M 2024-01-24T13:34:04.161 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:03 smithi029 ceph-mon[40651]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T13:34:04.161 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:03 smithi029 ceph-mon[40651]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-01-24T13:34:04.161 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:03 smithi029 ceph-mon[40651]: osdmap e56: 9 total, 8 up, 9 in 2024-01-24T13:34:04.161 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:03 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T13:34:04.161 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:03 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T13:34:04.161 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:03 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T13:34:04.161 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:03 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T13:34:04.161 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:03 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:04.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:03 smithi163 ceph-mon[44969]: Detected new or changed devices on smithi187 2024-01-24T13:34:04.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:03 smithi163 ceph-mon[44969]: Adjusting osd_memory_target on smithi187 to 5704M 2024-01-24T13:34:04.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:03 smithi163 ceph-mon[44969]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T13:34:04.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:03 smithi163 ceph-mon[44969]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-01-24T13:34:04.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:03 smithi163 ceph-mon[44969]: osdmap e56: 9 total, 8 up, 9 in 2024-01-24T13:34:04.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:03 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T13:34:04.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:03 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T13:34:04.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:03 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T13:34:04.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:03 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T13:34:04.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:03 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:04.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:03 smithi187 ceph-mon[44911]: Detected new or changed devices on smithi187 2024-01-24T13:34:04.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:03 smithi187 ceph-mon[44911]: Adjusting osd_memory_target on smithi187 to 5704M 2024-01-24T13:34:04.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:03 smithi187 ceph-mon[44911]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T13:34:04.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:03 smithi187 ceph-mon[44911]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-01-24T13:34:04.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:03 smithi187 ceph-mon[44911]: osdmap e56: 9 total, 8 up, 9 in 2024-01-24T13:34:04.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:03 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T13:34:04.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:03 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T13:34:04.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:03 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T13:34:04.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:03 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T13:34:04.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:03 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:05.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:04 smithi029 ceph-mon[40651]: purged_snaps scrub starts 2024-01-24T13:34:05.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:04 smithi029 ceph-mon[40651]: purged_snaps scrub ok 2024-01-24T13:34:05.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:04 smithi029 ceph-mon[40651]: from='client.14415 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:34:05.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:04 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T13:34:05.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:04 smithi029 ceph-mon[40651]: osd.8 [v2:172.21.15.187:6816/2856803882,v1:172.21.15.187:6817/2856803882] boot 2024-01-24T13:34:05.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:04 smithi029 ceph-mon[40651]: osdmap e57: 9 total, 9 up, 9 in 2024-01-24T13:34:05.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:04 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T13:34:05.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:04 smithi163 ceph-mon[44969]: purged_snaps scrub starts 2024-01-24T13:34:05.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:04 smithi163 ceph-mon[44969]: purged_snaps scrub ok 2024-01-24T13:34:05.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:04 smithi163 ceph-mon[44969]: from='client.14415 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:34:05.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:04 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T13:34:05.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:04 smithi163 ceph-mon[44969]: osd.8 [v2:172.21.15.187:6816/2856803882,v1:172.21.15.187:6817/2856803882] boot 2024-01-24T13:34:05.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:04 smithi163 ceph-mon[44969]: osdmap e57: 9 total, 9 up, 9 in 2024-01-24T13:34:05.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:04 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T13:34:05.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:04 smithi187 ceph-mon[44911]: purged_snaps scrub starts 2024-01-24T13:34:05.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:04 smithi187 ceph-mon[44911]: purged_snaps scrub ok 2024-01-24T13:34:05.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:04 smithi187 ceph-mon[44911]: from='client.14415 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:34:05.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:04 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T13:34:05.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:04 smithi187 ceph-mon[44911]: osd.8 [v2:172.21.15.187:6816/2856803882,v1:172.21.15.187:6817/2856803882] boot 2024-01-24T13:34:05.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:04 smithi187 ceph-mon[44911]: osdmap e57: 9 total, 9 up, 9 in 2024-01-24T13:34:05.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:04 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T13:34:06.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:05 smithi163 ceph-mon[44969]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T13:34:06.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:05 smithi163 ceph-mon[44969]: osdmap e58: 9 total, 9 up, 9 in 2024-01-24T13:34:06.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:05 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/3104589238' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cf417a8e-e576-43de-b89c-f104ee3969ca"}]: dispatch 2024-01-24T13:34:06.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:05 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/3104589238' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cf417a8e-e576-43de-b89c-f104ee3969ca"}]': finished 2024-01-24T13:34:06.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:05 smithi163 ceph-mon[44969]: osdmap e59: 10 total, 9 up, 10 in 2024-01-24T13:34:06.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:05 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T13:34:06.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:05 smithi163 ceph-mon[44969]: osdmap e60: 10 total, 9 up, 10 in 2024-01-24T13:34:06.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:05 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T13:34:06.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:05 smithi187 ceph-mon[44911]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T13:34:06.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:05 smithi187 ceph-mon[44911]: osdmap e58: 9 total, 9 up, 9 in 2024-01-24T13:34:06.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:05 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/3104589238' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cf417a8e-e576-43de-b89c-f104ee3969ca"}]: dispatch 2024-01-24T13:34:06.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:05 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/3104589238' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cf417a8e-e576-43de-b89c-f104ee3969ca"}]': finished 2024-01-24T13:34:06.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:05 smithi187 ceph-mon[44911]: osdmap e59: 10 total, 9 up, 10 in 2024-01-24T13:34:06.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:05 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T13:34:06.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:05 smithi187 ceph-mon[44911]: osdmap e60: 10 total, 9 up, 10 in 2024-01-24T13:34:06.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:05 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T13:34:06.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:05 smithi029 ceph-mon[40651]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T13:34:06.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:05 smithi029 ceph-mon[40651]: osdmap e58: 9 total, 9 up, 9 in 2024-01-24T13:34:06.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:05 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/3104589238' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cf417a8e-e576-43de-b89c-f104ee3969ca"}]: dispatch 2024-01-24T13:34:06.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:05 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/3104589238' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cf417a8e-e576-43de-b89c-f104ee3969ca"}]': finished 2024-01-24T13:34:06.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:05 smithi029 ceph-mon[40651]: osdmap e59: 10 total, 9 up, 10 in 2024-01-24T13:34:06.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:05 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T13:34:06.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:05 smithi029 ceph-mon[40651]: osdmap e60: 10 total, 9 up, 10 in 2024-01-24T13:34:06.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:05 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T13:34:07.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:06 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/2921589591' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T13:34:07.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:06 smithi163 ceph-mon[44969]: osdmap e61: 10 total, 9 up, 10 in 2024-01-24T13:34:07.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:06 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T13:34:07.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:06 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/2921589591' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T13:34:07.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:06 smithi187 ceph-mon[44911]: osdmap e61: 10 total, 9 up, 10 in 2024-01-24T13:34:07.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:06 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T13:34:07.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:06 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/2921589591' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T13:34:07.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:06 smithi029 ceph-mon[40651]: osdmap e61: 10 total, 9 up, 10 in 2024-01-24T13:34:07.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:06 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T13:34:08.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:07 smithi163 ceph-mon[44969]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2024-01-24T13:34:08.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:07 smithi187 ceph-mon[44911]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2024-01-24T13:34:08.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:07 smithi029 ceph-mon[40651]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2024-01-24T13:34:10.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:09 smithi029 ceph-mon[40651]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-01-24T13:34:10.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:09 smithi163 ceph-mon[44969]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-01-24T13:34:10.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:09 smithi187 ceph-mon[44911]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-01-24T13:34:12.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:11 smithi029 ceph-mon[40651]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-01-24T13:34:12.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:11 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-01-24T13:34:12.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:11 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:12.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:11 smithi163 ceph-mon[44969]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-01-24T13:34:12.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:11 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-01-24T13:34:12.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:11 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:12.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:11 smithi187 ceph-mon[44911]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-01-24T13:34:12.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:11 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-01-24T13:34:12.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:11 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:13.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:12 smithi163 ceph-mon[44969]: Deploying daemon osd.9 on smithi029 2024-01-24T13:34:13.295 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:12 smithi029 ceph-mon[40651]: Deploying daemon osd.9 on smithi029 2024-01-24T13:34:13.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:12 smithi187 ceph-mon[44911]: Deploying daemon osd.9 on smithi029 2024-01-24T13:34:14.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:13 smithi029 ceph-mon[40651]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-01-24T13:34:14.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:13 smithi163 ceph-mon[44969]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-01-24T13:34:14.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:13 smithi187 ceph-mon[44911]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-01-24T13:34:16.183 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:15 smithi029 ceph-mon[40651]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-01-24T13:34:16.183 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:15 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:16.183 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:15 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:16.183 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:15 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:34:16.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:15 smithi163 ceph-mon[44969]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-01-24T13:34:16.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:15 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:16.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:15 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:16.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:15 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:34:16.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:15 smithi187 ceph-mon[44911]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-01-24T13:34:16.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:15 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:16.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:15 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:16.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:15 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:34:17.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:17.999+0000 7f36c8ff9700 1 -- 172.21.15.29:0/1806075399 <== mgr.14152 v2:172.21.15.29:6800/2498991338 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f36cc068fd0 con 0x7f36b80a05a0 2024-01-24T13:34:18.000 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 9 on host 'smithi029' 2024-01-24T13:34:18.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:18.002+0000 7f36d1745700 1 -- 172.21.15.29:0/1806075399 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f36b80a05a0 msgr2=0x7f36b80a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:34:18.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:18.002+0000 7f36d1745700 1 --2- 172.21.15.29:0/1806075399 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f36b80a05a0 0x7f36b80a2a60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f36cc100d30 tx=0x7f36bc00a3b0 comp rx=0 tx=0).stop 2024-01-24T13:34:18.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:18.002+0000 7f36d1745700 1 -- 172.21.15.29:0/1806075399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36cc100ed0 msgr2=0x7f36cc1022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:34:18.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:18.002+0000 7f36d1745700 1 --2- 172.21.15.29:0/1806075399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36cc100ed0 0x7f36cc1022b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f36b4009990 tx=0x7f36b4004730 comp rx=0 tx=0).stop 2024-01-24T13:34:18.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:18.002+0000 7f36d1745700 1 -- 172.21.15.29:0/1806075399 shutdown_connections 2024-01-24T13:34:18.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:18.002+0000 7f36d1745700 1 --2- 172.21.15.29:0/1806075399 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36cc101870 0x7f36cc1027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:18.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:18.002+0000 7f36d1745700 1 --2- 172.21.15.29:0/1806075399 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f36cc0ffcd0 0x7f36cc103c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:18.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:18.002+0000 7f36d1745700 1 --2- 172.21.15.29:0/1806075399 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f36b80a05a0 0x7f36b80a2a60 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:18.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:18.002+0000 7f36d1745700 1 --2- 172.21.15.29:0/1806075399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36cc100ed0 0x7f36cc1022b0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:18.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:18.002+0000 7f36d1745700 1 -- 172.21.15.29:0/1806075399 >> 172.21.15.29:0/1806075399 conn(0x7f36cc0fb250 msgr2=0x7f36cc0688c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:34:18.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:18.002+0000 7f36d1745700 1 -- 172.21.15.29:0/1806075399 shutdown_connections 2024-01-24T13:34:18.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:18.002+0000 7f36d1745700 1 -- 172.21.15.29:0/1806075399 wait complete. 2024-01-24T13:34:18.361 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:18 smithi029 ceph-mon[40651]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-01-24T13:34:18.362 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:18 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:18.362 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:18 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:18.362 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:18 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:18.362 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:18 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:34:18.362 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:18 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:18.362 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:18 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:18.362 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:18 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:18.362 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:18 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:34:18.416 DEBUG:teuthology.orchestra.run.smithi029:osd.9> sudo journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@osd.9.service 2024-01-24T13:34:18.418 INFO:tasks.cephadm:Deploying osd.10 on smithi163 with /dev/vg_nvme/lv_1... 2024-01-24T13:34:18.418 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-01-24T13:34:18.457 INFO:journalctl@ceph.osd.9.smithi029.stdout:-- Logs begin at Wed 2024-01-24 13:13:42 UTC. -- 2024-01-24T13:34:18.464 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:18 smithi163 ceph-mon[44969]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-01-24T13:34:18.464 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:18 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:18.464 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:18 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:18.464 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:18 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:18.464 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:18 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:34:18.464 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:18 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:18.464 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:18 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:18.464 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:18 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:18.464 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:18 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:34:18.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:18 smithi187 ceph-mon[44911]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-01-24T13:34:18.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:18 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:18.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:18 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:18.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:18 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:18.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:18 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:34:18.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:18 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:18.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:18 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:18.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:18 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:18.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:18 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:34:18.663 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.b/config 2024-01-24T13:34:20.044 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-01-24T13:34:20.045 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-01-24T13:34:20.045 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10+0 records in 2024-01-24T13:34:20.045 INFO:teuthology.orchestra.run.smithi163.stderr:10+0 records out 2024-01-24T13:34:20.045 INFO:teuthology.orchestra.run.smithi163.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0161137 s, 651 MB/s 2024-01-24T13:34:20.045 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping successful for: 2024-01-24T13:34:20.099 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:34:19 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:34:19.658+0000 7fdcbcb78640 -1 osd.9 0 log_to_monitors true 2024-01-24T13:34:20.374 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:20 smithi163 ceph-mon[44969]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-01-24T13:34:20.374 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:20 smithi163 ceph-mon[44969]: from='osd.9 [v2:172.21.15.29:6826/1392813017,v1:172.21.15.29:6827/1392813017]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-01-24T13:34:20.393 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph orch daemon add osd smithi163:vg_nvme/lv_1 2024-01-24T13:34:20.477 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:20 smithi029 ceph-mon[40651]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-01-24T13:34:20.477 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:20 smithi029 ceph-mon[40651]: from='osd.9 [v2:172.21.15.29:6826/1392813017,v1:172.21.15.29:6827/1392813017]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-01-24T13:34:20.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:20 smithi187 ceph-mon[44911]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-01-24T13:34:20.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:20 smithi187 ceph-mon[44911]: from='osd.9 [v2:172.21.15.29:6826/1392813017,v1:172.21.15.29:6827/1392813017]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-01-24T13:34:20.612 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.b/config 2024-01-24T13:34:21.446 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:21 smithi163 ceph-mon[44969]: from='osd.9 [v2:172.21.15.29:6826/1392813017,v1:172.21.15.29:6827/1392813017]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-01-24T13:34:21.447 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:21 smithi163 ceph-mon[44969]: osdmap e62: 10 total, 9 up, 10 in 2024-01-24T13:34:21.447 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:21 smithi163 ceph-mon[44969]: from='osd.9 [v2:172.21.15.29:6826/1392813017,v1:172.21.15.29:6827/1392813017]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-01-24T13:34:21.447 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:21 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T13:34:21.447 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:21 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:21.447 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:21 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:21.447 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:21 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:21.447 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:21 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:21.447 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:21 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:21.447 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:21 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:21.447 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:21 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:21.447 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:21 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:21.447 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:21 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:34:21.447 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:21 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:21.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:21 smithi187 ceph-mon[44911]: from='osd.9 [v2:172.21.15.29:6826/1392813017,v1:172.21.15.29:6827/1392813017]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-01-24T13:34:21.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:21 smithi187 ceph-mon[44911]: osdmap e62: 10 total, 9 up, 10 in 2024-01-24T13:34:21.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:21 smithi187 ceph-mon[44911]: from='osd.9 [v2:172.21.15.29:6826/1392813017,v1:172.21.15.29:6827/1392813017]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-01-24T13:34:21.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:21 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T13:34:21.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:21 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:21.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:21 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:21.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:21 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:21.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:21 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:21.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:21 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:21.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:21 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:21.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:21 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:21.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:21 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:21.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:21 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:34:21.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:21 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:21.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:34:21 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:34:21.115+0000 7fdcb21dd700 -1 osd.9 0 waiting for initial osdmap 2024-01-24T13:34:21.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:34:21 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:34:21.136+0000 7fdcab7cd700 -1 osd.9 63 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-24T13:34:21.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:21 smithi029 ceph-mon[40651]: from='osd.9 [v2:172.21.15.29:6826/1392813017,v1:172.21.15.29:6827/1392813017]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-01-24T13:34:21.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:21 smithi029 ceph-mon[40651]: osdmap e62: 10 total, 9 up, 10 in 2024-01-24T13:34:21.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:21 smithi029 ceph-mon[40651]: from='osd.9 [v2:172.21.15.29:6826/1392813017,v1:172.21.15.29:6827/1392813017]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-01-24T13:34:21.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:21 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T13:34:21.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:21 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:21.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:21 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:21.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:21 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:21.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:21 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:21.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:21 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:21.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:21 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:21.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:21 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:21.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:21 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:21.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:21 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:34:21.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:21 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:21.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.622+0000 7efcdaddb700 1 -- 172.21.15.163:0/487131376 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efcd4100ed0 msgr2=0x7efcd4101330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:34:21.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.622+0000 7efcdaddb700 1 --2- 172.21.15.163:0/487131376 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efcd4100ed0 0x7efcd4101330 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efcc00099c0 tx=0x7efcc0009cd0 comp rx=0 tx=0).stop 2024-01-24T13:34:21.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.622+0000 7efcdaddb700 1 -- 172.21.15.163:0/487131376 shutdown_connections 2024-01-24T13:34:21.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.622+0000 7efcdaddb700 1 --2- 172.21.15.163:0/487131376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efcd4101870 0x7efcd4108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:21.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.622+0000 7efcdaddb700 1 --2- 172.21.15.163:0/487131376 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efcd40ffcd0 0x7efcd41000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:21.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.622+0000 7efcdaddb700 1 --2- 172.21.15.163:0/487131376 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efcd4100ed0 0x7efcd4101330 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:21.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.622+0000 7efcdaddb700 1 -- 172.21.15.163:0/487131376 >> 172.21.15.163:0/487131376 conn(0x7efcd40fb250 msgr2=0x7efcd40fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:34:21.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.622+0000 7efcdaddb700 1 -- 172.21.15.163:0/487131376 shutdown_connections 2024-01-24T13:34:21.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.622+0000 7efcdaddb700 1 -- 172.21.15.163:0/487131376 wait complete. 2024-01-24T13:34:21.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.623+0000 7efcdaddb700 1 Processor -- start 2024-01-24T13:34:21.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.623+0000 7efcdaddb700 1 -- start start 2024-01-24T13:34:21.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.624+0000 7efcdaddb700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efcd40ffcd0 0x7efcd4069c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:34:21.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.624+0000 7efcdaddb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efcd4100ed0 0x7efcd419b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:34:21.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.624+0000 7efcdaddb700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efcd4101870 0x7efcd419dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:34:21.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.624+0000 7efcdaddb700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd419e640 con 0x7efcd4100ed0 2024-01-24T13:34:21.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.624+0000 7efcdaddb700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd419e7b0 con 0x7efcd4101870 2024-01-24T13:34:21.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.624+0000 7efcdaddb700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd419e970 con 0x7efcd40ffcd0 2024-01-24T13:34:21.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.624+0000 7efcd9378700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efcd4101870 0x7efcd419dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:34:21.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.624+0000 7efcd9378700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efcd4101870 0x7efcd419dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:43020/0 (socket says 172.21.15.163:43020) 2024-01-24T13:34:21.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.624+0000 7efcd9378700 1 -- 172.21.15.163:0/3201727803 learned_addr learned my addr 172.21.15.163:0/3201727803 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-01-24T13:34:21.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.624+0000 7efcd8b77700 1 --2- 172.21.15.163:0/3201727803 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efcd40ffcd0 0x7efcd4069c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:34:21.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.624+0000 7efcd3fff700 1 --2- 172.21.15.163:0/3201727803 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efcd4100ed0 0x7efcd419b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:34:21.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.624+0000 7efcd9378700 1 -- 172.21.15.163:0/3201727803 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efcd40ffcd0 msgr2=0x7efcd4069c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:34:21.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.624+0000 7efcd9378700 1 --2- 172.21.15.163:0/3201727803 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efcd40ffcd0 0x7efcd4069c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:21.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.624+0000 7efcd9378700 1 -- 172.21.15.163:0/3201727803 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efcd4100ed0 msgr2=0x7efcd419b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:34:21.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.624+0000 7efcd9378700 1 --2- 172.21.15.163:0/3201727803 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efcd4100ed0 0x7efcd419b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:21.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.624+0000 7efcd9378700 1 -- 172.21.15.163:0/3201727803 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc00095f0 con 0x7efcd4101870 2024-01-24T13:34:21.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.624+0000 7efcd3fff700 1 --2- 172.21.15.163:0/3201727803 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efcd4100ed0 0x7efcd419b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T13:34:21.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.624+0000 7efcd9378700 1 --2- 172.21.15.163:0/3201727803 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efcd4101870 0x7efcd419dea0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efccc00b970 tx=0x7efccc00bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:34:21.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.624+0000 7efcd1ffb700 1 -- 172.21.15.163:0/3201727803 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efccc00c930 con 0x7efcd4101870 2024-01-24T13:34:21.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.624+0000 7efcd1ffb700 1 -- 172.21.15.163:0/3201727803 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efccc014440 con 0x7efcd4101870 2024-01-24T13:34:21.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.624+0000 7efcd1ffb700 1 -- 172.21.15.163:0/3201727803 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efccc0075d0 con 0x7efcd4101870 2024-01-24T13:34:21.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.624+0000 7efcdaddb700 1 -- 172.21.15.163:0/3201727803 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcd419ec50 con 0x7efcd4101870 2024-01-24T13:34:21.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.625+0000 7efcdaddb700 1 -- 172.21.15.163:0/3201727803 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcd419f0d0 con 0x7efcd4101870 2024-01-24T13:34:21.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.626+0000 7efcdaddb700 1 -- 172.21.15.163:0/3201727803 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcd41072f0 con 0x7efcd4101870 2024-01-24T13:34:21.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.627+0000 7efcd1ffb700 1 -- 172.21.15.163:0/3201727803 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7efccc0145b0 con 0x7efcd4101870 2024-01-24T13:34:21.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.627+0000 7efcd1ffb700 1 --2- 172.21.15.163:0/3201727803 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7efcc40a05a0 0x7efcc40a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:34:21.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.627+0000 7efcd1ffb700 1 -- 172.21.15.163:0/3201727803 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5981+0+0 (secure 0 0 0) 0x7efccc006e40 con 0x7efcd4101870 2024-01-24T13:34:21.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.627+0000 7efcd8b77700 1 --2- 172.21.15.163:0/3201727803 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7efcc40a05a0 0x7efcc40a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:34:21.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.628+0000 7efcd8b77700 1 --2- 172.21.15.163:0/3201727803 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7efcc40a05a0 0x7efcc40a2a60 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7efcd4100d30 tx=0x7efcc800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:34:21.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.632+0000 7efcd1ffb700 1 -- 172.21.15.163:0/3201727803 <== mon.2 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7efccc094b10 con 0x7efcd4101870 2024-01-24T13:34:21.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:21.827+0000 7efcdaddb700 1 -- 172.21.15.163:0/3201727803 --> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7efcd4061180 con 0x7efcc40a05a0 2024-01-24T13:34:22.375 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:22 smithi163 ceph-mon[44969]: Detected new or changed devices on smithi029 2024-01-24T13:34:22.375 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:22 smithi163 ceph-mon[44969]: Adjusting osd_memory_target on smithi029 to 4279M 2024-01-24T13:34:22.375 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:22 smithi163 ceph-mon[44969]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-01-24T13:34:22.375 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:22 smithi163 ceph-mon[44969]: from='osd.9 [v2:172.21.15.29:6826/1392813017,v1:172.21.15.29:6827/1392813017]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-01-24T13:34:22.375 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:22 smithi163 ceph-mon[44969]: osdmap e63: 10 total, 9 up, 10 in 2024-01-24T13:34:22.375 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:22 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T13:34:22.375 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:22 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T13:34:22.375 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:22 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T13:34:22.375 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:22 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T13:34:22.375 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:22 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:22.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:22 smithi187 ceph-mon[44911]: Detected new or changed devices on smithi029 2024-01-24T13:34:22.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:22 smithi187 ceph-mon[44911]: Adjusting osd_memory_target on smithi029 to 4279M 2024-01-24T13:34:22.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:22 smithi187 ceph-mon[44911]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-01-24T13:34:22.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:22 smithi187 ceph-mon[44911]: from='osd.9 [v2:172.21.15.29:6826/1392813017,v1:172.21.15.29:6827/1392813017]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-01-24T13:34:22.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:22 smithi187 ceph-mon[44911]: osdmap e63: 10 total, 9 up, 10 in 2024-01-24T13:34:22.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:22 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T13:34:22.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:22 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T13:34:22.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:22 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T13:34:22.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:22 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T13:34:22.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:22 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:22.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:22 smithi029 ceph-mon[40651]: Detected new or changed devices on smithi029 2024-01-24T13:34:22.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:22 smithi029 ceph-mon[40651]: Adjusting osd_memory_target on smithi029 to 4279M 2024-01-24T13:34:22.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:22 smithi029 ceph-mon[40651]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-01-24T13:34:22.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:22 smithi029 ceph-mon[40651]: from='osd.9 [v2:172.21.15.29:6826/1392813017,v1:172.21.15.29:6827/1392813017]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-01-24T13:34:22.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:22 smithi029 ceph-mon[40651]: osdmap e63: 10 total, 9 up, 10 in 2024-01-24T13:34:22.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:22 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T13:34:22.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:22 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T13:34:22.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:22 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T13:34:22.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:22 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T13:34:22.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:22 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:23.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:23 smithi163 ceph-mon[44969]: purged_snaps scrub starts 2024-01-24T13:34:23.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:23 smithi163 ceph-mon[44969]: purged_snaps scrub ok 2024-01-24T13:34:23.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:23 smithi163 ceph-mon[44969]: from='client.24368 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:34:23.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:23 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T13:34:23.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:23 smithi163 ceph-mon[44969]: osd.9 [v2:172.21.15.29:6826/1392813017,v1:172.21.15.29:6827/1392813017] boot 2024-01-24T13:34:23.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:23 smithi163 ceph-mon[44969]: osdmap e64: 10 total, 10 up, 10 in 2024-01-24T13:34:23.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:23 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T13:34:23.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:23 smithi187 ceph-mon[44911]: purged_snaps scrub starts 2024-01-24T13:34:23.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:23 smithi187 ceph-mon[44911]: purged_snaps scrub ok 2024-01-24T13:34:23.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:23 smithi187 ceph-mon[44911]: from='client.24368 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:34:23.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:23 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T13:34:23.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:23 smithi187 ceph-mon[44911]: osd.9 [v2:172.21.15.29:6826/1392813017,v1:172.21.15.29:6827/1392813017] boot 2024-01-24T13:34:23.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:23 smithi187 ceph-mon[44911]: osdmap e64: 10 total, 10 up, 10 in 2024-01-24T13:34:23.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:23 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T13:34:23.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:23 smithi029 ceph-mon[40651]: purged_snaps scrub starts 2024-01-24T13:34:23.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:23 smithi029 ceph-mon[40651]: purged_snaps scrub ok 2024-01-24T13:34:23.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:23 smithi029 ceph-mon[40651]: from='client.24368 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:34:23.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:23 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T13:34:23.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:23 smithi029 ceph-mon[40651]: osd.9 [v2:172.21.15.29:6826/1392813017,v1:172.21.15.29:6827/1392813017] boot 2024-01-24T13:34:23.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:23 smithi029 ceph-mon[40651]: osdmap e64: 10 total, 10 up, 10 in 2024-01-24T13:34:23.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:23 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T13:34:24.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:24 smithi163 ceph-mon[44969]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-01-24T13:34:24.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:24 smithi163 ceph-mon[44969]: osdmap e65: 10 total, 10 up, 10 in 2024-01-24T13:34:24.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:24 smithi163 ceph-mon[44969]: from='client.? 172.21.15.163:0/2830411590' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc7cfb9d-3bfa-45b2-929a-a8dc90f1a2a9"}]: dispatch 2024-01-24T13:34:24.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:24 smithi163 ceph-mon[44969]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc7cfb9d-3bfa-45b2-929a-a8dc90f1a2a9"}]: dispatch 2024-01-24T13:34:24.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:24 smithi163 ceph-mon[44969]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc7cfb9d-3bfa-45b2-929a-a8dc90f1a2a9"}]': finished 2024-01-24T13:34:24.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:24 smithi163 ceph-mon[44969]: osdmap e66: 11 total, 10 up, 11 in 2024-01-24T13:34:24.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:24 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-01-24T13:34:24.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:24 smithi187 ceph-mon[44911]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-01-24T13:34:24.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:24 smithi187 ceph-mon[44911]: osdmap e65: 10 total, 10 up, 10 in 2024-01-24T13:34:24.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:24 smithi187 ceph-mon[44911]: from='client.? 172.21.15.163:0/2830411590' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc7cfb9d-3bfa-45b2-929a-a8dc90f1a2a9"}]: dispatch 2024-01-24T13:34:24.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:24 smithi187 ceph-mon[44911]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc7cfb9d-3bfa-45b2-929a-a8dc90f1a2a9"}]: dispatch 2024-01-24T13:34:24.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:24 smithi187 ceph-mon[44911]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc7cfb9d-3bfa-45b2-929a-a8dc90f1a2a9"}]': finished 2024-01-24T13:34:24.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:24 smithi187 ceph-mon[44911]: osdmap e66: 11 total, 10 up, 11 in 2024-01-24T13:34:24.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:24 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-01-24T13:34:24.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:24 smithi029 ceph-mon[40651]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-01-24T13:34:24.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:24 smithi029 ceph-mon[40651]: osdmap e65: 10 total, 10 up, 10 in 2024-01-24T13:34:24.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:24 smithi029 ceph-mon[40651]: from='client.? 172.21.15.163:0/2830411590' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc7cfb9d-3bfa-45b2-929a-a8dc90f1a2a9"}]: dispatch 2024-01-24T13:34:24.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:24 smithi029 ceph-mon[40651]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc7cfb9d-3bfa-45b2-929a-a8dc90f1a2a9"}]: dispatch 2024-01-24T13:34:24.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:24 smithi029 ceph-mon[40651]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc7cfb9d-3bfa-45b2-929a-a8dc90f1a2a9"}]': finished 2024-01-24T13:34:24.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:24 smithi029 ceph-mon[40651]: osdmap e66: 11 total, 10 up, 11 in 2024-01-24T13:34:24.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:24 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-01-24T13:34:25.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:25 smithi163 ceph-mon[44969]: from='client.? 172.21.15.163:0/1508634447' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T13:34:25.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:25 smithi187 ceph-mon[44911]: from='client.? 172.21.15.163:0/1508634447' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T13:34:25.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:25 smithi029 ceph-mon[40651]: from='client.? 172.21.15.163:0/1508634447' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T13:34:26.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:26 smithi163 ceph-mon[44969]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-01-24T13:34:26.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:26 smithi187 ceph-mon[44911]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-01-24T13:34:26.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:26 smithi029 ceph-mon[40651]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-01-24T13:34:28.327 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:28 smithi163 ceph-mon[44969]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-01-24T13:34:28.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:28 smithi187 ceph-mon[44911]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-01-24T13:34:28.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:28 smithi029 ceph-mon[40651]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-01-24T13:34:30.405 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:30 smithi163 ceph-mon[44969]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-01-24T13:34:30.406 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:30 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-01-24T13:34:30.406 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:30 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:30.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:30 smithi187 ceph-mon[44911]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-01-24T13:34:30.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:30 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-01-24T13:34:30.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:30 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:30.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:30 smithi029 ceph-mon[40651]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-01-24T13:34:30.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:30 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-01-24T13:34:30.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:30 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:31.288 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:31 smithi163 ceph-mon[44969]: Deploying daemon osd.10 on smithi163 2024-01-24T13:34:31.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:31 smithi187 ceph-mon[44911]: Deploying daemon osd.10 on smithi163 2024-01-24T13:34:31.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:31 smithi029 ceph-mon[40651]: Deploying daemon osd.10 on smithi163 2024-01-24T13:34:32.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:32 smithi163 ceph-mon[44969]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-01-24T13:34:32.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:32 smithi187 ceph-mon[44911]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-01-24T13:34:32.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:32 smithi029 ceph-mon[40651]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-01-24T13:34:34.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:34 smithi163 ceph-mon[44969]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-01-24T13:34:34.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:34 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:34.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:34 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:34.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:34 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:34:34.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:34 smithi187 ceph-mon[44911]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-01-24T13:34:34.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:34 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:34.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:34 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:34.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:34 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:34:34.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:34 smithi029 ceph-mon[40651]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-01-24T13:34:34.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:34 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:34.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:34 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:34.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:34 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:34:35.631 INFO:teuthology.orchestra.run.smithi163.stdout:Created osd(s) 10 on host 'smithi163' 2024-01-24T13:34:35.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:35.631+0000 7efcd1ffb700 1 -- 172.21.15.163:0/3201727803 <== mgr.14152 v2:172.21.15.29:6800/2498991338 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7efcd4061180 con 0x7efcc40a05a0 2024-01-24T13:34:35.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:35.633+0000 7efcdaddb700 1 -- 172.21.15.163:0/3201727803 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7efcc40a05a0 msgr2=0x7efcc40a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:34:35.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:35.633+0000 7efcdaddb700 1 --2- 172.21.15.163:0/3201727803 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7efcc40a05a0 0x7efcc40a2a60 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7efcd4100d30 tx=0x7efcc800b410 comp rx=0 tx=0).stop 2024-01-24T13:34:35.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:35.633+0000 7efcdaddb700 1 -- 172.21.15.163:0/3201727803 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efcd4101870 msgr2=0x7efcd419dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:34:35.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:35.633+0000 7efcdaddb700 1 --2- 172.21.15.163:0/3201727803 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efcd4101870 0x7efcd419dea0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efccc00b970 tx=0x7efccc00bc80 comp rx=0 tx=0).stop 2024-01-24T13:34:35.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:35.633+0000 7efcdaddb700 1 -- 172.21.15.163:0/3201727803 shutdown_connections 2024-01-24T13:34:35.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:35.633+0000 7efcdaddb700 1 --2- 172.21.15.163:0/3201727803 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efcd4101870 0x7efcd419dea0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:35.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:35.633+0000 7efcdaddb700 1 --2- 172.21.15.163:0/3201727803 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efcd40ffcd0 0x7efcd4069c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:35.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:35.633+0000 7efcdaddb700 1 --2- 172.21.15.163:0/3201727803 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7efcc40a05a0 0x7efcc40a2a60 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:35.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:35.633+0000 7efcdaddb700 1 --2- 172.21.15.163:0/3201727803 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efcd4100ed0 0x7efcd419b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:35.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:35.633+0000 7efcdaddb700 1 -- 172.21.15.163:0/3201727803 >> 172.21.15.163:0/3201727803 conn(0x7efcd40fb250 msgr2=0x7efcd4104430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:34:35.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:35.633+0000 7efcdaddb700 1 -- 172.21.15.163:0/3201727803 shutdown_connections 2024-01-24T13:34:35.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-01-24T13:34:35.633+0000 7efcdaddb700 1 -- 172.21.15.163:0/3201727803 wait complete. 2024-01-24T13:34:36.037 DEBUG:teuthology.orchestra.run.smithi163:osd.10> sudo journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@osd.10.service 2024-01-24T13:34:36.039 INFO:tasks.cephadm:Deploying osd.11 on smithi187 with /dev/vg_nvme/lv_1... 2024-01-24T13:34:36.040 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-01-24T13:34:36.071 INFO:journalctl@ceph.osd.10.smithi163.stdout:-- Logs begin at Wed 2024-01-24 13:14:09 UTC. -- 2024-01-24T13:34:36.184 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:35 smithi163 ceph-mon[44969]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-01-24T13:34:36.184 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:35 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:36.184 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:35 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:36.184 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:35 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:36.184 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:35 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:34:36.184 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:35 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:36.185 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:35 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:36.185 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:35 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:36.185 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:35 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:34:36.272 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.c/config 2024-01-24T13:34:36.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:35 smithi187 ceph-mon[44911]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-01-24T13:34:36.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:35 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:36.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:35 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:36.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:35 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:36.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:35 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:34:36.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:35 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:36.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:35 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:36.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:35 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:36.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:35 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:34:36.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:35 smithi029 ceph-mon[40651]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-01-24T13:34:36.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:35 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:36.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:35 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:36.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:35 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:36.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:35 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:34:36.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:35 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:36.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:35 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:36.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:35 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:36.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:35 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:34:37.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:34:37 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:34:37.201+0000 7f264f730640 -1 osd.10 0 log_to_monitors true 2024-01-24T13:34:37.550 INFO:teuthology.orchestra.run.smithi187.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-01-24T13:34:37.550 INFO:teuthology.orchestra.run.smithi187.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-01-24T13:34:37.551 INFO:teuthology.orchestra.run.smithi187.stderr: stderr: 10+0 records in 2024-01-24T13:34:37.551 INFO:teuthology.orchestra.run.smithi187.stderr:10+0 records out 2024-01-24T13:34:37.551 INFO:teuthology.orchestra.run.smithi187.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0114845 s, 913 MB/s 2024-01-24T13:34:37.551 INFO:teuthology.orchestra.run.smithi187.stderr:--> Zapping successful for: 2024-01-24T13:34:37.927 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph orch daemon add osd smithi187:vg_nvme/lv_1 2024-01-24T13:34:38.149 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.c/config 2024-01-24T13:34:38.162 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:37 smithi163 ceph-mon[44969]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-01-24T13:34:38.163 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:37 smithi163 ceph-mon[44969]: from='osd.10 [v2:172.21.15.163:6824/2523735721,v1:172.21.15.163:6825/2523735721]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-01-24T13:34:38.163 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:37 smithi163 ceph-mon[44969]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-01-24T13:34:38.247 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:37 smithi187 ceph-mon[44911]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-01-24T13:34:38.247 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:37 smithi187 ceph-mon[44911]: from='osd.10 [v2:172.21.15.163:6824/2523735721,v1:172.21.15.163:6825/2523735721]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-01-24T13:34:38.247 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:37 smithi187 ceph-mon[44911]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-01-24T13:34:38.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:37 smithi029 ceph-mon[40651]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-01-24T13:34:38.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:37 smithi029 ceph-mon[40651]: from='osd.10 [v2:172.21.15.163:6824/2523735721,v1:172.21.15.163:6825/2523735721]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-01-24T13:34:38.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:37 smithi029 ceph-mon[40651]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-01-24T13:34:39.100 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.100+0000 7f383207c700 1 -- 172.21.15.187:0/1866523995 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f382c106b20 msgr2=0x7f382c108f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:34:39.101 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.100+0000 7f383207c700 1 --2- 172.21.15.187:0/1866523995 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f382c106b20 0x7f382c108f30 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f382400b0a0 tx=0x7f382400b3b0 comp rx=0 tx=0).stop 2024-01-24T13:34:39.101 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.100+0000 7f383207c700 1 -- 172.21.15.187:0/1866523995 shutdown_connections 2024-01-24T13:34:39.101 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.100+0000 7f383207c700 1 --2- 172.21.15.187:0/1866523995 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f382c106b20 0x7f382c108f30 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:39.101 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.100+0000 7f383207c700 1 --2- 172.21.15.187:0/1866523995 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f382c1018c0 0x7f382c103cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:39.101 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.100+0000 7f383207c700 1 --2- 172.21.15.187:0/1866523995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f382c1041f0 0x7f382c1065e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:39.101 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.100+0000 7f383207c700 1 -- 172.21.15.187:0/1866523995 >> 172.21.15.187:0/1866523995 conn(0x7f382c0fb250 msgr2=0x7f382c0fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:34:39.101 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.100+0000 7f383207c700 1 -- 172.21.15.187:0/1866523995 shutdown_connections 2024-01-24T13:34:39.101 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.100+0000 7f383207c700 1 -- 172.21.15.187:0/1866523995 wait complete. 2024-01-24T13:34:39.103 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.100+0000 7f383207c700 1 Processor -- start 2024-01-24T13:34:39.103 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.101+0000 7f383207c700 1 -- start start 2024-01-24T13:34:39.103 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.101+0000 7f383207c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f382c1018c0 0x7f382c196b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:34:39.103 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.101+0000 7f383207c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f382c1041f0 0x7f382c1970c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:34:39.103 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.101+0000 7f383207c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f382c106b20 0x7f382c19e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:34:39.103 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.101+0000 7f383207c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382c19e710 con 0x7f382c1041f0 2024-01-24T13:34:39.104 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.101+0000 7f383207c700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382c19e850 con 0x7f382c1018c0 2024-01-24T13:34:39.104 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.101+0000 7f383207c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382c19e9e0 con 0x7f382c106b20 2024-01-24T13:34:39.104 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.101+0000 7f382bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f382c106b20 0x7f382c19e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:34:39.104 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.101+0000 7f382bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f382c106b20 0x7f382c19e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:34248/0 (socket says 172.21.15.187:34248) 2024-01-24T13:34:39.104 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.101+0000 7f382bfff700 1 -- 172.21.15.187:0/4125174860 learned_addr learned my addr 172.21.15.187:0/4125174860 (peer_addr_for_me v2:172.21.15.187:0/0) 2024-01-24T13:34:39.104 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.101+0000 7f382b7fe700 1 --2- 172.21.15.187:0/4125174860 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f382c1018c0 0x7f382c196b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:34:39.104 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.101+0000 7f382affd700 1 --2- 172.21.15.187:0/4125174860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f382c1041f0 0x7f382c1970c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:34:39.104 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.101+0000 7f382bfff700 1 -- 172.21.15.187:0/4125174860 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f382c1018c0 msgr2=0x7f382c196b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:34:39.104 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.101+0000 7f382bfff700 1 --2- 172.21.15.187:0/4125174860 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f382c1018c0 0x7f382c196b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:39.104 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.101+0000 7f382bfff700 1 -- 172.21.15.187:0/4125174860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f382c1041f0 msgr2=0x7f382c1970c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:34:39.104 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.101+0000 7f382bfff700 1 --2- 172.21.15.187:0/4125174860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f382c1041f0 0x7f382c1970c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:39.104 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.101+0000 7f382bfff700 1 -- 172.21.15.187:0/4125174860 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3824009d00 con 0x7f382c106b20 2024-01-24T13:34:39.105 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.101+0000 7f382affd700 1 --2- 172.21.15.187:0/4125174860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f382c1041f0 0x7f382c1970c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T13:34:39.105 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.102+0000 7f382bfff700 1 --2- 172.21.15.187:0/4125174860 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f382c106b20 0x7f382c19e140 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3824006070 tx=0x7f3824004d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:34:39.105 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.102+0000 7f3828ff9700 1 -- 172.21.15.187:0/4125174860 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382400e040 con 0x7f382c106b20 2024-01-24T13:34:39.105 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.102+0000 7f383207c700 1 -- 172.21.15.187:0/4125174860 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f382c19ec60 con 0x7f382c106b20 2024-01-24T13:34:39.105 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.102+0000 7f3828ff9700 1 -- 172.21.15.187:0/4125174860 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f382400cea0 con 0x7f382c106b20 2024-01-24T13:34:39.105 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.102+0000 7f3828ff9700 1 -- 172.21.15.187:0/4125174860 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3824004020 con 0x7f382c106b20 2024-01-24T13:34:39.105 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.102+0000 7f383207c700 1 -- 172.21.15.187:0/4125174860 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f382c19f120 con 0x7f382c106b20 2024-01-24T13:34:39.106 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.103+0000 7f383207c700 1 -- 172.21.15.187:0/4125174860 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f382c0fcdd0 con 0x7f382c106b20 2024-01-24T13:34:39.106 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.104+0000 7f3828ff9700 1 -- 172.21.15.187:0/4125174860 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f382401d690 con 0x7f382c106b20 2024-01-24T13:34:39.106 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.104+0000 7f3828ff9700 1 --2- 172.21.15.187:0/4125174860 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f38180a0520 0x7f38180a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:34:39.106 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.105+0000 7f3828ff9700 1 -- 172.21.15.187:0/4125174860 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f38240cca50 con 0x7f382c106b20 2024-01-24T13:34:39.106 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.105+0000 7f382b7fe700 1 --2- 172.21.15.187:0/4125174860 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f38180a0520 0x7f38180a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:34:39.106 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.105+0000 7f382b7fe700 1 --2- 172.21.15.187:0/4125174860 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f38180a0520 0x7f38180a29e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3814005950 tx=0x7f381400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:34:39.109 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.109+0000 7f3828ff9700 1 -- 172.21.15.187:0/4125174860 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f3824096d50 con 0x7f382c106b20 2024-01-24T13:34:39.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:39 smithi187 ceph-mon[44911]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-01-24T13:34:39.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:39 smithi187 ceph-mon[44911]: osdmap e67: 11 total, 10 up, 11 in 2024-01-24T13:34:39.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:39 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-01-24T13:34:39.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:39 smithi187 ceph-mon[44911]: from='osd.10 [v2:172.21.15.163:6824/2523735721,v1:172.21.15.163:6825/2523735721]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-01-24T13:34:39.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:39 smithi187 ceph-mon[44911]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-01-24T13:34:39.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:39 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:39.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:39 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:39.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:39 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:39.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:39 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:39.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:39 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:39.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:39 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:39.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:39 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:39.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:39 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:39.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:39 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:34:39.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:39 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:39.306 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:39.306+0000 7f383207c700 1 -- 172.21.15.187:0/4125174860 --> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f382c061180 con 0x7f38180a0520 2024-01-24T13:34:39.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:39 smithi029 ceph-mon[40651]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-01-24T13:34:39.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:39 smithi029 ceph-mon[40651]: osdmap e67: 11 total, 10 up, 11 in 2024-01-24T13:34:39.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:39 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-01-24T13:34:39.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:39 smithi029 ceph-mon[40651]: from='osd.10 [v2:172.21.15.163:6824/2523735721,v1:172.21.15.163:6825/2523735721]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-01-24T13:34:39.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:39 smithi029 ceph-mon[40651]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-01-24T13:34:39.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:39 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:39.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:39 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:39.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:39 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:39.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:39 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:39.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:39 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:39.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:39 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:39.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:39 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:39.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:39 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:39.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:39 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:34:39.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:39 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:39.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:39 smithi163 ceph-mon[44969]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-01-24T13:34:39.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:39 smithi163 ceph-mon[44969]: osdmap e67: 11 total, 10 up, 11 in 2024-01-24T13:34:39.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:39 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-01-24T13:34:39.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:39 smithi163 ceph-mon[44969]: from='osd.10 [v2:172.21.15.163:6824/2523735721,v1:172.21.15.163:6825/2523735721]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-01-24T13:34:39.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:39 smithi163 ceph-mon[44969]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-01-24T13:34:39.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:39 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:39.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:39 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:39.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:39 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:39.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:39 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:39.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:39 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:39.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:39 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:39.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:39 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:39.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:39 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:39.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:39 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:34:39.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:39 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:39.484 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:34:39 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:34:39.006+0000 7f2644d95700 -1 osd.10 0 waiting for initial osdmap 2024-01-24T13:34:39.484 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:34:39 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:34:39.020+0000 7f263e385700 -1 osd.10 68 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-24T13:34:40.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:40 smithi187 ceph-mon[44911]: Detected new or changed devices on smithi163 2024-01-24T13:34:40.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:40 smithi187 ceph-mon[44911]: Adjusting osd_memory_target on smithi163 to 4278M 2024-01-24T13:34:40.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:40 smithi187 ceph-mon[44911]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-01-24T13:34:40.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:40 smithi187 ceph-mon[44911]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-01-24T13:34:40.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:40 smithi187 ceph-mon[44911]: osdmap e68: 11 total, 10 up, 11 in 2024-01-24T13:34:40.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:40 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-01-24T13:34:40.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:40 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-01-24T13:34:40.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:40 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T13:34:40.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:40 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T13:34:40.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:40 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:40.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:40 smithi029 ceph-mon[40651]: Detected new or changed devices on smithi163 2024-01-24T13:34:40.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:40 smithi029 ceph-mon[40651]: Adjusting osd_memory_target on smithi163 to 4278M 2024-01-24T13:34:40.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:40 smithi029 ceph-mon[40651]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-01-24T13:34:40.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:40 smithi029 ceph-mon[40651]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-01-24T13:34:40.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:40 smithi029 ceph-mon[40651]: osdmap e68: 11 total, 10 up, 11 in 2024-01-24T13:34:40.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:40 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-01-24T13:34:40.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:40 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-01-24T13:34:40.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:40 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T13:34:40.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:40 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T13:34:40.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:40 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:40.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:40 smithi163 ceph-mon[44969]: Detected new or changed devices on smithi163 2024-01-24T13:34:40.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:40 smithi163 ceph-mon[44969]: Adjusting osd_memory_target on smithi163 to 4278M 2024-01-24T13:34:40.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:40 smithi163 ceph-mon[44969]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-01-24T13:34:40.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:40 smithi163 ceph-mon[44969]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-01-24T13:34:40.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:40 smithi163 ceph-mon[44969]: osdmap e68: 11 total, 10 up, 11 in 2024-01-24T13:34:40.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:40 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-01-24T13:34:40.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:40 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-01-24T13:34:40.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:40 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T13:34:40.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:40 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T13:34:40.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:40 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:41.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:41 smithi187 ceph-mon[44911]: purged_snaps scrub starts 2024-01-24T13:34:41.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:41 smithi187 ceph-mon[44911]: purged_snaps scrub ok 2024-01-24T13:34:41.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:41 smithi187 ceph-mon[44911]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:34:41.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:41 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-01-24T13:34:41.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:41 smithi187 ceph-mon[44911]: osd.10 [v2:172.21.15.163:6824/2523735721,v1:172.21.15.163:6825/2523735721] boot 2024-01-24T13:34:41.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:41 smithi187 ceph-mon[44911]: osdmap e69: 11 total, 11 up, 11 in 2024-01-24T13:34:41.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:41 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-01-24T13:34:41.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:41 smithi187 ceph-mon[44911]: osdmap e70: 11 total, 11 up, 11 in 2024-01-24T13:34:41.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:41 smithi029 ceph-mon[40651]: purged_snaps scrub starts 2024-01-24T13:34:41.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:41 smithi029 ceph-mon[40651]: purged_snaps scrub ok 2024-01-24T13:34:41.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:41 smithi029 ceph-mon[40651]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:34:41.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:41 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-01-24T13:34:41.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:41 smithi029 ceph-mon[40651]: osd.10 [v2:172.21.15.163:6824/2523735721,v1:172.21.15.163:6825/2523735721] boot 2024-01-24T13:34:41.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:41 smithi029 ceph-mon[40651]: osdmap e69: 11 total, 11 up, 11 in 2024-01-24T13:34:41.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:41 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-01-24T13:34:41.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:41 smithi029 ceph-mon[40651]: osdmap e70: 11 total, 11 up, 11 in 2024-01-24T13:34:41.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:41 smithi163 ceph-mon[44969]: purged_snaps scrub starts 2024-01-24T13:34:41.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:41 smithi163 ceph-mon[44969]: purged_snaps scrub ok 2024-01-24T13:34:41.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:41 smithi163 ceph-mon[44969]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:34:41.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:41 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-01-24T13:34:41.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:41 smithi163 ceph-mon[44969]: osd.10 [v2:172.21.15.163:6824/2523735721,v1:172.21.15.163:6825/2523735721] boot 2024-01-24T13:34:41.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:41 smithi163 ceph-mon[44969]: osdmap e69: 11 total, 11 up, 11 in 2024-01-24T13:34:41.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:41 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-01-24T13:34:41.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:41 smithi163 ceph-mon[44969]: osdmap e70: 11 total, 11 up, 11 in 2024-01-24T13:34:42.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:42 smithi187 ceph-mon[44911]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-01-24T13:34:42.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:42 smithi187 ceph-mon[44911]: from='client.? 172.21.15.187:0/813663220' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1453684e-fe11-4cbc-a868-cf90b3a7a021"}]: dispatch 2024-01-24T13:34:42.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:42 smithi187 ceph-mon[44911]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1453684e-fe11-4cbc-a868-cf90b3a7a021"}]: dispatch 2024-01-24T13:34:42.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:42 smithi187 ceph-mon[44911]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1453684e-fe11-4cbc-a868-cf90b3a7a021"}]': finished 2024-01-24T13:34:42.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:42 smithi187 ceph-mon[44911]: osdmap e71: 12 total, 11 up, 12 in 2024-01-24T13:34:42.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:42 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T13:34:42.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:42 smithi187 ceph-mon[44911]: from='client.? 172.21.15.187:0/1662346371' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T13:34:42.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:42 smithi029 ceph-mon[40651]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-01-24T13:34:42.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:42 smithi029 ceph-mon[40651]: from='client.? 172.21.15.187:0/813663220' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1453684e-fe11-4cbc-a868-cf90b3a7a021"}]: dispatch 2024-01-24T13:34:42.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:42 smithi029 ceph-mon[40651]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1453684e-fe11-4cbc-a868-cf90b3a7a021"}]: dispatch 2024-01-24T13:34:42.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:42 smithi029 ceph-mon[40651]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1453684e-fe11-4cbc-a868-cf90b3a7a021"}]': finished 2024-01-24T13:34:42.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:42 smithi029 ceph-mon[40651]: osdmap e71: 12 total, 11 up, 12 in 2024-01-24T13:34:42.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:42 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T13:34:42.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:42 smithi029 ceph-mon[40651]: from='client.? 172.21.15.187:0/1662346371' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T13:34:42.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:42 smithi163 ceph-mon[44969]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-01-24T13:34:42.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:42 smithi163 ceph-mon[44969]: from='client.? 172.21.15.187:0/813663220' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1453684e-fe11-4cbc-a868-cf90b3a7a021"}]: dispatch 2024-01-24T13:34:42.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:42 smithi163 ceph-mon[44969]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1453684e-fe11-4cbc-a868-cf90b3a7a021"}]: dispatch 2024-01-24T13:34:42.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:42 smithi163 ceph-mon[44969]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1453684e-fe11-4cbc-a868-cf90b3a7a021"}]': finished 2024-01-24T13:34:42.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:42 smithi163 ceph-mon[44969]: osdmap e71: 12 total, 11 up, 12 in 2024-01-24T13:34:42.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:42 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T13:34:42.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:42 smithi163 ceph-mon[44969]: from='client.? 172.21.15.187:0/1662346371' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T13:34:43.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:43 smithi029 ceph-mon[40651]: osdmap e72: 12 total, 11 up, 12 in 2024-01-24T13:34:43.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:43 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T13:34:43.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:43 smithi163 ceph-mon[44969]: osdmap e72: 12 total, 11 up, 12 in 2024-01-24T13:34:43.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:43 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T13:34:43.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:43 smithi187 ceph-mon[44911]: osdmap e72: 12 total, 11 up, 12 in 2024-01-24T13:34:43.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:43 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T13:34:44.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:44 smithi029 ceph-mon[40651]: pgmap v188: 1 pgs: 1 peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 152 KiB/s, 0 objects/s recovering 2024-01-24T13:34:44.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:44 smithi029 ceph-mon[40651]: osdmap e73: 12 total, 11 up, 12 in 2024-01-24T13:34:44.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:44 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T13:34:44.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:44 smithi163 ceph-mon[44969]: pgmap v188: 1 pgs: 1 peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 152 KiB/s, 0 objects/s recovering 2024-01-24T13:34:44.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:44 smithi163 ceph-mon[44969]: osdmap e73: 12 total, 11 up, 12 in 2024-01-24T13:34:44.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:44 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T13:34:44.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:44 smithi187 ceph-mon[44911]: pgmap v188: 1 pgs: 1 peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 152 KiB/s, 0 objects/s recovering 2024-01-24T13:34:44.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:44 smithi187 ceph-mon[44911]: osdmap e73: 12 total, 11 up, 12 in 2024-01-24T13:34:44.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:44 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T13:34:46.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:46 smithi029 ceph-mon[40651]: pgmap v190: 1 pgs: 1 peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-01-24T13:34:46.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:46 smithi163 ceph-mon[44969]: pgmap v190: 1 pgs: 1 peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-01-24T13:34:46.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:46 smithi187 ceph-mon[44911]: pgmap v190: 1 pgs: 1 peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-01-24T13:34:47.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:47 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-01-24T13:34:47.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:47 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:47.440 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:47 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-01-24T13:34:47.440 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:47 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:47.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:47 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-01-24T13:34:47.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:47 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:48.251 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:48 smithi187 ceph-mon[44911]: pgmap v191: 1 pgs: 1 peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-01-24T13:34:48.252 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:48 smithi187 ceph-mon[44911]: Deploying daemon osd.11 on smithi187 2024-01-24T13:34:48.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:48 smithi029 ceph-mon[40651]: pgmap v191: 1 pgs: 1 peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-01-24T13:34:48.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:48 smithi029 ceph-mon[40651]: Deploying daemon osd.11 on smithi187 2024-01-24T13:34:48.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:48 smithi163 ceph-mon[44969]: pgmap v191: 1 pgs: 1 peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-01-24T13:34:48.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:48 smithi163 ceph-mon[44969]: Deploying daemon osd.11 on smithi187 2024-01-24T13:34:50.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:50 smithi029 ceph-mon[40651]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-01-24T13:34:50.368 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:50 smithi187 ceph-mon[44911]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-01-24T13:34:50.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:50 smithi163 ceph-mon[44969]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-01-24T13:34:51.532 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:51 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:51.532 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:51 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:51.532 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:51 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:34:51.532 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:51 smithi187 ceph-mon[44911]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-01-24T13:34:51.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:51 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:51.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:51 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:51.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:51 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:34:51.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:51 smithi163 ceph-mon[44969]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-01-24T13:34:51.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:51 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:51.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:51 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:51.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:51 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:34:51.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:51 smithi029 ceph-mon[40651]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-01-24T13:34:52.510 INFO:teuthology.orchestra.run.smithi187.stdout:Created osd(s) 11 on host 'smithi187' 2024-01-24T13:34:52.510 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:52.509+0000 7f3828ff9700 1 -- 172.21.15.187:0/4125174860 <== mgr.14152 v2:172.21.15.29:6800/2498991338 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f382c061180 con 0x7f38180a0520 2024-01-24T13:34:52.512 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:52.511+0000 7f383207c700 1 -- 172.21.15.187:0/4125174860 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f38180a0520 msgr2=0x7f38180a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:34:52.512 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:52.511+0000 7f383207c700 1 --2- 172.21.15.187:0/4125174860 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f38180a0520 0x7f38180a29e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3814005950 tx=0x7f381400b410 comp rx=0 tx=0).stop 2024-01-24T13:34:52.512 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:52.511+0000 7f383207c700 1 -- 172.21.15.187:0/4125174860 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f382c106b20 msgr2=0x7f382c19e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:34:52.512 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:52.511+0000 7f383207c700 1 --2- 172.21.15.187:0/4125174860 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f382c106b20 0x7f382c19e140 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3824006070 tx=0x7f3824004d00 comp rx=0 tx=0).stop 2024-01-24T13:34:52.512 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:52.511+0000 7f383207c700 1 -- 172.21.15.187:0/4125174860 shutdown_connections 2024-01-24T13:34:52.512 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:52.511+0000 7f383207c700 1 --2- 172.21.15.187:0/4125174860 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f382c106b20 0x7f382c19e140 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:52.512 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:52.511+0000 7f383207c700 1 --2- 172.21.15.187:0/4125174860 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f382c1018c0 0x7f382c196b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:52.512 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:52.511+0000 7f383207c700 1 --2- 172.21.15.187:0/4125174860 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f38180a0520 0x7f38180a29e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:52.512 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:52.511+0000 7f383207c700 1 --2- 172.21.15.187:0/4125174860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f382c1041f0 0x7f382c1970c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:52.512 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:52.511+0000 7f383207c700 1 -- 172.21.15.187:0/4125174860 >> 172.21.15.187:0/4125174860 conn(0x7f382c0fb250 msgr2=0x7f382c1025d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:34:52.512 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:52.511+0000 7f383207c700 1 -- 172.21.15.187:0/4125174860 shutdown_connections 2024-01-24T13:34:52.512 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:34:52.511+0000 7f383207c700 1 -- 172.21.15.187:0/4125174860 wait complete. 2024-01-24T13:34:53.024 DEBUG:teuthology.orchestra.run.smithi187:osd.11> sudo journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@osd.11.service 2024-01-24T13:34:53.026 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-01-24T13:34:53.027 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph osd stat -f json 2024-01-24T13:34:53.057 INFO:journalctl@ceph.osd.11.smithi187.stdout:-- Logs begin at Wed 2024-01-24 13:13:23 UTC. -- 2024-01-24T13:34:53.288 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T13:34:53.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:53 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:53.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:53 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:53.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:53 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:53.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:53 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:34:53.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:53 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:53.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:53 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:53.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:53 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:53.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:53 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:34:53.386 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:53 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:53.386 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:53 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:53.386 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:53 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:53.386 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:53 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:34:53.386 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:53 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:53.386 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:53 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:53.386 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:53 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:53.386 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:53 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:34:53.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:53 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:53.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:53 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:53.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:53 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:53.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:53 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:34:53.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:53 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:53.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:53 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:53.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:53 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:53.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:53 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:34:54.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.323+0000 7fd8aa8ed700 1 -- 172.21.15.29:0/2404342981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8a4104190 msgr2=0x7fd8a4106580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:34:54.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.323+0000 7fd8aa8ed700 1 --2- 172.21.15.29:0/2404342981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8a4104190 0x7fd8a4106580 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd898009990 tx=0x7fd898009ca0 comp rx=0 tx=0).stop 2024-01-24T13:34:54.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.323+0000 7fd8aa8ed700 1 -- 172.21.15.29:0/2404342981 shutdown_connections 2024-01-24T13:34:54.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.323+0000 7fd8aa8ed700 1 --2- 172.21.15.29:0/2404342981 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd8a4106ac0 0x7fd8a4108f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:54.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.323+0000 7fd8aa8ed700 1 --2- 172.21.15.29:0/2404342981 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8a4101860 0x7fd8a4103c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:54.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.323+0000 7fd8aa8ed700 1 --2- 172.21.15.29:0/2404342981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8a4104190 0x7fd8a4106580 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:54.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.323+0000 7fd8aa8ed700 1 -- 172.21.15.29:0/2404342981 >> 172.21.15.29:0/2404342981 conn(0x7fd8a40fb250 msgr2=0x7fd8a40fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:34:54.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.323+0000 7fd8aa8ed700 1 -- 172.21.15.29:0/2404342981 shutdown_connections 2024-01-24T13:34:54.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.323+0000 7fd8aa8ed700 1 -- 172.21.15.29:0/2404342981 wait complete. 2024-01-24T13:34:54.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.323+0000 7fd8aa8ed700 1 Processor -- start 2024-01-24T13:34:54.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.323+0000 7fd8aa8ed700 1 -- start start 2024-01-24T13:34:54.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.324+0000 7fd8aa8ed700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8a4101860 0x7fd8a4196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:34:54.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.324+0000 7fd8aa8ed700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8a4104190 0x7fd8a4196f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:34:54.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.324+0000 7fd8aa8ed700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd8a4106ac0 0x7fd8a419dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:34:54.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.324+0000 7fd8aa8ed700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a419e590 con 0x7fd8a4101860 2024-01-24T13:34:54.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.324+0000 7fd8aa8ed700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a419e6d0 con 0x7fd8a4106ac0 2024-01-24T13:34:54.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.324+0000 7fd8aa8ed700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a419e860 con 0x7fd8a4104190 2024-01-24T13:34:54.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.324+0000 7fd8a3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8a4101860 0x7fd8a4196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:34:54.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.324+0000 7fd8a3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8a4101860 0x7fd8a4196a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39812/0 (socket says 172.21.15.29:39812) 2024-01-24T13:34:54.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.324+0000 7fd8a3fff700 1 -- 172.21.15.29:0/1200562951 learned_addr learned my addr 172.21.15.29:0/1200562951 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:34:54.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.324+0000 7fd8a37fe700 1 --2- 172.21.15.29:0/1200562951 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8a4104190 0x7fd8a4196f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:34:54.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.324+0000 7fd8a8e8a700 1 --2- 172.21.15.29:0/1200562951 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd8a4106ac0 0x7fd8a419dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:34:54.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.324+0000 7fd8a3fff700 1 -- 172.21.15.29:0/1200562951 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8a4104190 msgr2=0x7fd8a4196f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:34:54.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.324+0000 7fd8a3fff700 1 --2- 172.21.15.29:0/1200562951 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8a4104190 0x7fd8a4196f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:54.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.324+0000 7fd8a3fff700 1 -- 172.21.15.29:0/1200562951 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd8a4106ac0 msgr2=0x7fd8a419dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:34:54.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.324+0000 7fd8a3fff700 1 --2- 172.21.15.29:0/1200562951 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd8a4106ac0 0x7fd8a419dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:54.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.324+0000 7fd8a3fff700 1 -- 172.21.15.29:0/1200562951 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8980095f0 con 0x7fd8a4101860 2024-01-24T13:34:54.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.324+0000 7fd8a37fe700 1 --2- 172.21.15.29:0/1200562951 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8a4104190 0x7fd8a4196f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T13:34:54.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.324+0000 7fd8a3fff700 1 --2- 172.21.15.29:0/1200562951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8a4101860 0x7fd8a4196a00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd89400e9d0 tx=0x7fd89400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:34:54.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.325+0000 7fd8a17fa700 1 -- 172.21.15.29:0/1200562951 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd894009790 con 0x7fd8a4101860 2024-01-24T13:34:54.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.325+0000 7fd8a17fa700 1 -- 172.21.15.29:0/1200562951 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd89400cbd0 con 0x7fd8a4101860 2024-01-24T13:34:54.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.325+0000 7fd8aa8ed700 1 -- 172.21.15.29:0/1200562951 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8a419eb40 con 0x7fd8a4101860 2024-01-24T13:34:54.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.325+0000 7fd8a17fa700 1 -- 172.21.15.29:0/1200562951 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd894018680 con 0x7fd8a4101860 2024-01-24T13:34:54.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.325+0000 7fd8aa8ed700 1 -- 172.21.15.29:0/1200562951 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8a419f040 con 0x7fd8a4101860 2024-01-24T13:34:54.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.326+0000 7fd8aa8ed700 1 -- 172.21.15.29:0/1200562951 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8a40fcdd0 con 0x7fd8a4101860 2024-01-24T13:34:54.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.327+0000 7fd8a17fa700 1 -- 172.21.15.29:0/1200562951 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7fd89400cd40 con 0x7fd8a4101860 2024-01-24T13:34:54.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.328+0000 7fd8a17fa700 1 --2- 172.21.15.29:0/1200562951 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fd88c0a0520 0x7fd88c0a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:34:54.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.328+0000 7fd8a37fe700 1 --2- 172.21.15.29:0/1200562951 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fd88c0a0520 0x7fd88c0a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:34:54.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.328+0000 7fd8a17fa700 1 -- 172.21.15.29:0/1200562951 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fd894014070 con 0x7fd8a4101860 2024-01-24T13:34:54.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.328+0000 7fd8a37fe700 1 --2- 172.21.15.29:0/1200562951 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fd88c0a0520 0x7fd88c0a29e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd898006010 tx=0x7fd89800b280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:34:54.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.331+0000 7fd8a17fa700 1 -- 172.21.15.29:0/1200562951 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fd894095ad0 con 0x7fd8a4101860 2024-01-24T13:34:54.425 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:54 smithi187 ceph-mon[44911]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-01-24T13:34:54.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:54 smithi163 ceph-mon[44969]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-01-24T13:34:54.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.539+0000 7fd8aa8ed700 1 -- 172.21.15.29:0/1200562951 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fd8a402cc70 con 0x7fd8a4101860 2024-01-24T13:34:54.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.539+0000 7fd8a17fa700 1 -- 172.21.15.29:0/1200562951 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v73) v1 ==== 74+0+133 (secure 0 0 0) 0x7fd8940990f0 con 0x7fd8a4101860 2024-01-24T13:34:54.540 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:34:54.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.542+0000 7fd8aa8ed700 1 -- 172.21.15.29:0/1200562951 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fd88c0a0520 msgr2=0x7fd88c0a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:34:54.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.542+0000 7fd8aa8ed700 1 --2- 172.21.15.29:0/1200562951 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fd88c0a0520 0x7fd88c0a29e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd898006010 tx=0x7fd89800b280 comp rx=0 tx=0).stop 2024-01-24T13:34:54.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.542+0000 7fd8aa8ed700 1 -- 172.21.15.29:0/1200562951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8a4101860 msgr2=0x7fd8a4196a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:34:54.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.542+0000 7fd8aa8ed700 1 --2- 172.21.15.29:0/1200562951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8a4101860 0x7fd8a4196a00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd89400e9d0 tx=0x7fd89400ece0 comp rx=0 tx=0).stop 2024-01-24T13:34:54.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.542+0000 7fd8aa8ed700 1 -- 172.21.15.29:0/1200562951 shutdown_connections 2024-01-24T13:34:54.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.542+0000 7fd8aa8ed700 1 --2- 172.21.15.29:0/1200562951 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd8a4106ac0 0x7fd8a419dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:54.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.542+0000 7fd8aa8ed700 1 --2- 172.21.15.29:0/1200562951 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fd88c0a0520 0x7fd88c0a29e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:54.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.542+0000 7fd8aa8ed700 1 --2- 172.21.15.29:0/1200562951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8a4101860 0x7fd8a4196a00 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:54.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.542+0000 7fd8aa8ed700 1 --2- 172.21.15.29:0/1200562951 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8a4104190 0x7fd8a4196f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:54.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.542+0000 7fd8aa8ed700 1 -- 172.21.15.29:0/1200562951 >> 172.21.15.29:0/1200562951 conn(0x7fd8a40fb250 msgr2=0x7fd8a41024d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:34:54.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.542+0000 7fd8aa8ed700 1 -- 172.21.15.29:0/1200562951 shutdown_connections 2024-01-24T13:34:54.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:54.542+0000 7fd8aa8ed700 1 -- 172.21.15.29:0/1200562951 wait complete. 2024-01-24T13:34:54.574 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:54 smithi029 ceph-mon[40651]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-01-24T13:34:54.735 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:34:54 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:34:54.423+0000 7fe8507db640 -1 osd.11 0 log_to_monitors true 2024-01-24T13:34:54.926 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":73,"num_osds":12,"num_up_osds":11,"osd_up_since":1706103280,"num_in_osds":12,"osd_in_since":1706103281,"num_remapped_pgs":0} 2024-01-24T13:34:55.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:55 smithi029 ceph-mon[40651]: from='osd.11 [v2:172.21.15.187:6824/1828122314,v1:172.21.15.187:6825/1828122314]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-01-24T13:34:55.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:55 smithi029 ceph-mon[40651]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-01-24T13:34:55.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:55 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/1200562951' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-24T13:34:55.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:55 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:55.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:55 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:55.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:55 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:55.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:55 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:55.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:55 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:55.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:55 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:55.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:55 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:55.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:55 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:55.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:55 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:34:55.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:55 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:55.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:55 smithi163 ceph-mon[44969]: from='osd.11 [v2:172.21.15.187:6824/1828122314,v1:172.21.15.187:6825/1828122314]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-01-24T13:34:55.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:55 smithi163 ceph-mon[44969]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-01-24T13:34:55.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:55 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/1200562951' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-24T13:34:55.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:55 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:55.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:55 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:55.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:55 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:55.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:55 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:55.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:55 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:55.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:55 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:55.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:55 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:55.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:55 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:55.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:55 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:34:55.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:55 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:55.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:55 smithi187 ceph-mon[44911]: from='osd.11 [v2:172.21.15.187:6824/1828122314,v1:172.21.15.187:6825/1828122314]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-01-24T13:34:55.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:55 smithi187 ceph-mon[44911]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-01-24T13:34:55.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:55 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/1200562951' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-24T13:34:55.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:55 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:55.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:55 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:55.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:55 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:55.556 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:55 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:55.556 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:55 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:55.556 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:55 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-01-24T13:34:55.556 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:55 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:55.556 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:55 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:34:55.556 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:55 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:34:55.556 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:55 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:34:55.927 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph osd stat -f json 2024-01-24T13:34:56.184 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T13:34:56.289 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:56 smithi029 ceph-mon[40651]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-01-24T13:34:56.289 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:56 smithi029 ceph-mon[40651]: Detected new or changed devices on smithi187 2024-01-24T13:34:56.290 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:56 smithi029 ceph-mon[40651]: Adjusting osd_memory_target on smithi187 to 4278M 2024-01-24T13:34:56.290 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:56 smithi029 ceph-mon[40651]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-01-24T13:34:56.290 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:56 smithi029 ceph-mon[40651]: osdmap e74: 12 total, 11 up, 12 in 2024-01-24T13:34:56.290 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:56 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T13:34:56.290 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:56 smithi029 ceph-mon[40651]: from='osd.11 [v2:172.21.15.187:6824/1828122314,v1:172.21.15.187:6825/1828122314]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-01-24T13:34:56.290 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:56 smithi029 ceph-mon[40651]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-01-24T13:34:56.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:56 smithi163 ceph-mon[44969]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-01-24T13:34:56.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:56 smithi163 ceph-mon[44969]: Detected new or changed devices on smithi187 2024-01-24T13:34:56.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:56 smithi163 ceph-mon[44969]: Adjusting osd_memory_target on smithi187 to 4278M 2024-01-24T13:34:56.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:56 smithi163 ceph-mon[44969]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-01-24T13:34:56.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:56 smithi163 ceph-mon[44969]: osdmap e74: 12 total, 11 up, 12 in 2024-01-24T13:34:56.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:56 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T13:34:56.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:56 smithi163 ceph-mon[44969]: from='osd.11 [v2:172.21.15.187:6824/1828122314,v1:172.21.15.187:6825/1828122314]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-01-24T13:34:56.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:56 smithi163 ceph-mon[44969]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-01-24T13:34:56.554 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:34:56 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:34:56.145+0000 7fe845e40700 -1 osd.11 0 waiting for initial osdmap 2024-01-24T13:34:56.555 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:34:56 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:34:56.163+0000 7fe840432700 -1 osd.11 75 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-24T13:34:56.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:56 smithi187 ceph-mon[44911]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-01-24T13:34:56.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:56 smithi187 ceph-mon[44911]: Detected new or changed devices on smithi187 2024-01-24T13:34:56.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:56 smithi187 ceph-mon[44911]: Adjusting osd_memory_target on smithi187 to 4278M 2024-01-24T13:34:56.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:56 smithi187 ceph-mon[44911]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-01-24T13:34:56.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:56 smithi187 ceph-mon[44911]: osdmap e74: 12 total, 11 up, 12 in 2024-01-24T13:34:56.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:56 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T13:34:56.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:56 smithi187 ceph-mon[44911]: from='osd.11 [v2:172.21.15.187:6824/1828122314,v1:172.21.15.187:6825/1828122314]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-01-24T13:34:56.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:56 smithi187 ceph-mon[44911]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2024-01-24T13:34:57.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.147+0000 7fcde2352700 1 -- 172.21.15.29:0/1836865161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcddc101860 msgr2=0x7fcddc103c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:34:57.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.147+0000 7fcde2352700 1 --2- 172.21.15.29:0/1836865161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcddc101860 0x7fcddc103c50 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc009990 tx=0x7fcdcc009ca0 comp rx=0 tx=0).stop 2024-01-24T13:34:57.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.147+0000 7fcde2352700 1 -- 172.21.15.29:0/1836865161 shutdown_connections 2024-01-24T13:34:57.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.147+0000 7fcde2352700 1 --2- 172.21.15.29:0/1836865161 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcddc106ac0 0x7fcddc108f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:57.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.147+0000 7fcde2352700 1 --2- 172.21.15.29:0/1836865161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcddc101860 0x7fcddc103c50 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:57.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.147+0000 7fcde2352700 1 --2- 172.21.15.29:0/1836865161 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcddc104190 0x7fcddc106580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:57.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.147+0000 7fcde2352700 1 -- 172.21.15.29:0/1836865161 >> 172.21.15.29:0/1836865161 conn(0x7fcddc0fb250 msgr2=0x7fcddc0fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:34:57.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.147+0000 7fcde2352700 1 -- 172.21.15.29:0/1836865161 shutdown_connections 2024-01-24T13:34:57.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.147+0000 7fcde2352700 1 -- 172.21.15.29:0/1836865161 wait complete. 2024-01-24T13:34:57.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.148+0000 7fcde2352700 1 Processor -- start 2024-01-24T13:34:57.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.148+0000 7fcde2352700 1 -- start start 2024-01-24T13:34:57.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.148+0000 7fcde2352700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcddc101860 0x7fcddc1011c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:34:57.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.148+0000 7fcde2352700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcddc104190 0x7fcddc0ff810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:34:57.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.148+0000 7fcde2352700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcddc106ac0 0x7fcddc0ffd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:34:57.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.148+0000 7fcde2352700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcddc1004d0 con 0x7fcddc104190 2024-01-24T13:34:57.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.148+0000 7fcde2352700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcddc100640 con 0x7fcddc101860 2024-01-24T13:34:57.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.148+0000 7fcde2352700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcddc1a0b40 con 0x7fcddc106ac0 2024-01-24T13:34:57.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.148+0000 7fcddb7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcddc104190 0x7fcddc0ff810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:34:57.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.148+0000 7fcddbfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcddc101860 0x7fcddc1011c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:34:57.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.148+0000 7fcddb7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcddc104190 0x7fcddc0ff810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39826/0 (socket says 172.21.15.29:39826) 2024-01-24T13:34:57.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.148+0000 7fcddb7fe700 1 -- 172.21.15.29:0/996840070 learned_addr learned my addr 172.21.15.29:0/996840070 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:34:57.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.148+0000 7fcddbfff700 1 --2- 172.21.15.29:0/996840070 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcddc101860 0x7fcddc1011c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.29:43266/0 (socket says 172.21.15.29:43266) 2024-01-24T13:34:57.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.149+0000 7fcde08ef700 1 --2- 172.21.15.29:0/996840070 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcddc106ac0 0x7fcddc0ffd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:34:57.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.149+0000 7fcddbfff700 1 -- 172.21.15.29:0/996840070 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcddc106ac0 msgr2=0x7fcddc0ffd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:34:57.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.149+0000 7fcddbfff700 1 --2- 172.21.15.29:0/996840070 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcddc106ac0 0x7fcddc0ffd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:57.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.149+0000 7fcddbfff700 1 -- 172.21.15.29:0/996840070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcddc104190 msgr2=0x7fcddc0ff810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:34:57.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.149+0000 7fcddbfff700 1 --2- 172.21.15.29:0/996840070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcddc104190 0x7fcddc0ff810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:57.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.149+0000 7fcddbfff700 1 -- 172.21.15.29:0/996840070 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdcc0095f0 con 0x7fcddc101860 2024-01-24T13:34:57.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.149+0000 7fcddb7fe700 1 --2- 172.21.15.29:0/996840070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcddc104190 0x7fcddc0ff810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T13:34:57.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.149+0000 7fcddbfff700 1 --2- 172.21.15.29:0/996840070 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcddc101860 0x7fcddc1011c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc00b680 tx=0x7fcdcc009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:34:57.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.149+0000 7fcdd97fa700 1 -- 172.21.15.29:0/996840070 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc01d070 con 0x7fcddc101860 2024-01-24T13:34:57.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.149+0000 7fcdd97fa700 1 -- 172.21.15.29:0/996840070 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcdcc021410 con 0x7fcddc101860 2024-01-24T13:34:57.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.149+0000 7fcdd97fa700 1 -- 172.21.15.29:0/996840070 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc017650 con 0x7fcddc101860 2024-01-24T13:34:57.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.149+0000 7fcde2352700 1 -- 172.21.15.29:0/996840070 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcddc1a0dc0 con 0x7fcddc101860 2024-01-24T13:34:57.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.150+0000 7fcde2352700 1 -- 172.21.15.29:0/996840070 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcddc1a1180 con 0x7fcddc101860 2024-01-24T13:34:57.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.150+0000 7fcde2352700 1 -- 172.21.15.29:0/996840070 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcddc0fcdd0 con 0x7fcddc101860 2024-01-24T13:34:57.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.152+0000 7fcdd97fa700 1 -- 172.21.15.29:0/996840070 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7fcdcc021810 con 0x7fcddc101860 2024-01-24T13:34:57.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.153+0000 7fcdd97fa700 1 --2- 172.21.15.29:0/996840070 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fcdc40a05a0 0x7fcdc40a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:34:57.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.153+0000 7fcddb7fe700 1 --2- 172.21.15.29:0/996840070 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fcdc40a05a0 0x7fcdc40a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:34:57.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.153+0000 7fcdd97fa700 1 -- 172.21.15.29:0/996840070 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fcdcc0ccdf0 con 0x7fcddc101860 2024-01-24T13:34:57.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.153+0000 7fcddb7fe700 1 --2- 172.21.15.29:0/996840070 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fcdc40a05a0 0x7fcdc40a2a60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fcdd0011550 tx=0x7fcdd0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:34:57.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.156+0000 7fcdd97fa700 1 -- 172.21.15.29:0/996840070 <== mon.2 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fcdcc096ec0 con 0x7fcddc101860 2024-01-24T13:34:57.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:57 smithi029 ceph-mon[40651]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-01-24T13:34:57.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:57 smithi029 ceph-mon[40651]: osdmap e75: 12 total, 11 up, 12 in 2024-01-24T13:34:57.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:57 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T13:34:57.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:57 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T13:34:57.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.363+0000 7fcde2352700 1 -- 172.21.15.29:0/996840070 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fcddc10c630 con 0x7fcddc101860 2024-01-24T13:34:57.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.363+0000 7fcdd97fa700 1 -- 172.21.15.29:0/996840070 <== mon.2 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v76) v1 ==== 74+0+133 (secure 0 0 0) 0x7fcdcc09a4e0 con 0x7fcddc101860 2024-01-24T13:34:57.364 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:34:57.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.366+0000 7fcde2352700 1 -- 172.21.15.29:0/996840070 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fcdc40a05a0 msgr2=0x7fcdc40a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:34:57.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.366+0000 7fcde2352700 1 --2- 172.21.15.29:0/996840070 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fcdc40a05a0 0x7fcdc40a2a60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fcdd0011550 tx=0x7fcdd0009260 comp rx=0 tx=0).stop 2024-01-24T13:34:57.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.366+0000 7fcde2352700 1 -- 172.21.15.29:0/996840070 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcddc101860 msgr2=0x7fcddc1011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:34:57.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.366+0000 7fcde2352700 1 --2- 172.21.15.29:0/996840070 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcddc101860 0x7fcddc1011c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc00b680 tx=0x7fcdcc009580 comp rx=0 tx=0).stop 2024-01-24T13:34:57.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.366+0000 7fcde2352700 1 -- 172.21.15.29:0/996840070 shutdown_connections 2024-01-24T13:34:57.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.366+0000 7fcde2352700 1 --2- 172.21.15.29:0/996840070 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcddc106ac0 0x7fcddc0ffd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:57.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.366+0000 7fcde2352700 1 --2- 172.21.15.29:0/996840070 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcddc101860 0x7fcddc1011c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:57.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.366+0000 7fcde2352700 1 --2- 172.21.15.29:0/996840070 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fcdc40a05a0 0x7fcdc40a2a60 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:57.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.366+0000 7fcde2352700 1 --2- 172.21.15.29:0/996840070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcddc104190 0x7fcddc0ff810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:57.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.366+0000 7fcde2352700 1 -- 172.21.15.29:0/996840070 >> 172.21.15.29:0/996840070 conn(0x7fcddc0fb250 msgr2=0x7fcddc1024d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:34:57.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.366+0000 7fcde2352700 1 -- 172.21.15.29:0/996840070 shutdown_connections 2024-01-24T13:34:57.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:57.366+0000 7fcde2352700 1 -- 172.21.15.29:0/996840070 wait complete. 2024-01-24T13:34:57.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:57 smithi163 ceph-mon[44969]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-01-24T13:34:57.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:57 smithi163 ceph-mon[44969]: osdmap e75: 12 total, 11 up, 12 in 2024-01-24T13:34:57.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:57 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T13:34:57.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:57 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T13:34:57.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:57 smithi187 ceph-mon[44911]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-01-24T13:34:57.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:57 smithi187 ceph-mon[44911]: osdmap e75: 12 total, 11 up, 12 in 2024-01-24T13:34:57.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:57 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T13:34:57.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:57 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T13:34:57.743 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":76,"num_osds":12,"num_up_osds":12,"osd_up_since":1706103297,"num_in_osds":12,"osd_in_since":1706103281,"num_remapped_pgs":0} 2024-01-24T13:34:57.743 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph osd dump --format=json 2024-01-24T13:34:57.998 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T13:34:58.456 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:58 smithi029 ceph-mon[40651]: purged_snaps scrub starts 2024-01-24T13:34:58.456 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:58 smithi029 ceph-mon[40651]: purged_snaps scrub ok 2024-01-24T13:34:58.456 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:58 smithi029 ceph-mon[40651]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-01-24T13:34:58.456 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:58 smithi029 ceph-mon[40651]: osd.11 [v2:172.21.15.187:6824/1828122314,v1:172.21.15.187:6825/1828122314] boot 2024-01-24T13:34:58.456 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:58 smithi029 ceph-mon[40651]: osdmap e76: 12 total, 12 up, 12 in 2024-01-24T13:34:58.456 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:58 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T13:34:58.456 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:58 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/996840070' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-24T13:34:58.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:58 smithi163 ceph-mon[44969]: purged_snaps scrub starts 2024-01-24T13:34:58.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:58 smithi163 ceph-mon[44969]: purged_snaps scrub ok 2024-01-24T13:34:58.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:58 smithi163 ceph-mon[44969]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-01-24T13:34:58.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:58 smithi163 ceph-mon[44969]: osd.11 [v2:172.21.15.187:6824/1828122314,v1:172.21.15.187:6825/1828122314] boot 2024-01-24T13:34:58.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:58 smithi163 ceph-mon[44969]: osdmap e76: 12 total, 12 up, 12 in 2024-01-24T13:34:58.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:58 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T13:34:58.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:58 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/996840070' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-24T13:34:58.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:58 smithi187 ceph-mon[44911]: purged_snaps scrub starts 2024-01-24T13:34:58.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:58 smithi187 ceph-mon[44911]: purged_snaps scrub ok 2024-01-24T13:34:58.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:58 smithi187 ceph-mon[44911]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-01-24T13:34:58.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:58 smithi187 ceph-mon[44911]: osd.11 [v2:172.21.15.187:6824/1828122314,v1:172.21.15.187:6825/1828122314] boot 2024-01-24T13:34:58.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:58 smithi187 ceph-mon[44911]: osdmap e76: 12 total, 12 up, 12 in 2024-01-24T13:34:58.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:58 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T13:34:58.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:58 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/996840070' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-24T13:34:59.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.010+0000 7f9807caf700 1 -- 172.21.15.29:0/3589308155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9800103cd0 msgr2=0x7f9800104150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:34:59.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.010+0000 7f9807caf700 1 --2- 172.21.15.29:0/3589308155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9800103cd0 0x7f9800104150 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f97f0009990 tx=0x7f97f0009ca0 comp rx=0 tx=0).stop 2024-01-24T13:34:59.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.010+0000 7f9807caf700 1 -- 172.21.15.29:0/3589308155 shutdown_connections 2024-01-24T13:34:59.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.010+0000 7f9807caf700 1 --2- 172.21.15.29:0/3589308155 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9800068860 0x7f9800068ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:59.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.010+0000 7f9807caf700 1 --2- 172.21.15.29:0/3589308155 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9800105d00 0x7f9800103790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:59.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.010+0000 7f9807caf700 1 --2- 172.21.15.29:0/3589308155 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9800103cd0 0x7f9800104150 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:59.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.010+0000 7f9807caf700 1 -- 172.21.15.29:0/3589308155 >> 172.21.15.29:0/3589308155 conn(0x7f9800076d40 msgr2=0x7f9800077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:34:59.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.010+0000 7f9807caf700 1 -- 172.21.15.29:0/3589308155 shutdown_connections 2024-01-24T13:34:59.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.010+0000 7f9807caf700 1 -- 172.21.15.29:0/3589308155 wait complete. 2024-01-24T13:34:59.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.011+0000 7f9807caf700 1 Processor -- start 2024-01-24T13:34:59.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.011+0000 7f9807caf700 1 -- start start 2024-01-24T13:34:59.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.011+0000 7f9807caf700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9800068860 0x7f980019d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:34:59.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.011+0000 7f9807caf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9800103cd0 0x7f980019da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:34:59.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.011+0000 7f9807caf700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9800105d00 0x7f98001a1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:34:59.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.011+0000 7f9807caf700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98001a2480 con 0x7f9800068860 2024-01-24T13:34:59.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.011+0000 7f9807caf700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98001a25f0 con 0x7f9800105d00 2024-01-24T13:34:59.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.011+0000 7f9807caf700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98001a27b0 con 0x7f9800103cd0 2024-01-24T13:34:59.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.011+0000 7f9805a4b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9800068860 0x7f980019d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:34:59.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.011+0000 7f9805a4b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9800068860 0x7f980019d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39840/0 (socket says 172.21.15.29:39840) 2024-01-24T13:34:59.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.011+0000 7f9805a4b700 1 -- 172.21.15.29:0/3493522125 learned_addr learned my addr 172.21.15.29:0/3493522125 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:34:59.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.012+0000 7f980624c700 1 --2- 172.21.15.29:0/3493522125 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9800105d00 0x7f98001a1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:34:59.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.012+0000 7f980524a700 1 --2- 172.21.15.29:0/3493522125 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9800103cd0 0x7f980019da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:34:59.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.012+0000 7f9805a4b700 1 -- 172.21.15.29:0/3493522125 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9800103cd0 msgr2=0x7f980019da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:34:59.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.012+0000 7f9805a4b700 1 --2- 172.21.15.29:0/3493522125 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9800103cd0 0x7f980019da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:59.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.012+0000 7f9805a4b700 1 -- 172.21.15.29:0/3493522125 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9800105d00 msgr2=0x7f98001a1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:34:59.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.012+0000 7f9805a4b700 1 --2- 172.21.15.29:0/3493522125 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9800105d00 0x7f98001a1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:59.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.012+0000 7f9805a4b700 1 -- 172.21.15.29:0/3493522125 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f00095f0 con 0x7f9800068860 2024-01-24T13:34:59.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.012+0000 7f9805a4b700 1 --2- 172.21.15.29:0/3493522125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9800068860 0x7f980019d4c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f97fc00d950 tx=0x7f97fc00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:34:59.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.012+0000 7f97f6ffd700 1 -- 172.21.15.29:0/3493522125 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97fc009940 con 0x7f9800068860 2024-01-24T13:34:59.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.012+0000 7f97f6ffd700 1 -- 172.21.15.29:0/3493522125 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f97fc010460 con 0x7f9800068860 2024-01-24T13:34:59.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.012+0000 7f9807caf700 1 -- 172.21.15.29:0/3493522125 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98001a2a90 con 0x7f9800068860 2024-01-24T13:34:59.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.012+0000 7f97f6ffd700 1 -- 172.21.15.29:0/3493522125 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97fc00f620 con 0x7f9800068860 2024-01-24T13:34:59.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.012+0000 7f9807caf700 1 -- 172.21.15.29:0/3493522125 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98001aa2f0 con 0x7f9800068860 2024-01-24T13:34:59.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.013+0000 7f9807caf700 1 -- 172.21.15.29:0/3493522125 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f980004ea40 con 0x7f9800068860 2024-01-24T13:34:59.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.014+0000 7f97f6ffd700 1 -- 172.21.15.29:0/3493522125 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f97fc0108d0 con 0x7f9800068860 2024-01-24T13:34:59.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.015+0000 7f97f6ffd700 1 --2- 172.21.15.29:0/3493522125 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f97ec0a0520 0x7f97ec0a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:34:59.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.015+0000 7f980524a700 1 --2- 172.21.15.29:0/3493522125 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f97ec0a0520 0x7f97ec0a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:34:59.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.015+0000 7f97f6ffd700 1 -- 172.21.15.29:0/3493522125 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f97fc0cb300 con 0x7f9800068860 2024-01-24T13:34:59.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.016+0000 7f980524a700 1 --2- 172.21.15.29:0/3493522125 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f97ec0a0520 0x7f97ec0a29e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f97f0006010 tx=0x7f97f000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:34:59.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.019+0000 7f97f6ffd700 1 -- 172.21.15.29:0/3493522125 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f97fc095340 con 0x7f9800068860 2024-01-24T13:34:59.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.225+0000 7f9807caf700 1 -- 172.21.15.29:0/3493522125 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98001aa4b0 con 0x7f9800068860 2024-01-24T13:34:59.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.226+0000 7f97f6ffd700 1 -- 172.21.15.29:0/3493522125 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+18443 (secure 0 0 0) 0x7f97fc098960 con 0x7f9800068860 2024-01-24T13:34:59.227 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:34:59.227 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":77,"fsid":"83afcf5e-babc-11ee-95b1-87774f69a715","created":"2024-01-24T13:29:33.175895+0000","modified":"2024-01-24T13:34:58.148429+0000","last_up_change":"2024-01-24T13:34:57.141902+0000","last_in_change":"2024-01-24T13:34:41.071943+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T13:32:21.264634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9857a607-2e6f-41a5-a86a-795970be6210","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":2406629076},{"type":"v1","addr":"172.21.15.29:6803","nonce":2406629076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":2406629076},{"type":"v1","addr":"172.21.15.29:6805","nonce":2406629076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2406629076},{"type":"v1","addr":"172.21.15.29:6809","nonce":2406629076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":2406629076},{"type":"v1","addr":"172.21.15.29:6807","nonce":2406629076}]},"public_addr":"172.21.15.29:6803/2406629076","cluster_addr":"172.21.15.29:6805/2406629076","heartbeat_back_addr":"172.21.15.29:6809/2406629076","heartbeat_front_addr":"172.21.15.29:6807/2406629076","state":["exists","up"]},{"osd":1,"uuid":"8bb2ba48-23d4-4bbd-a912-ae74f9e7f6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":2217244566},{"type":"v1","addr":"172.21.15.163:6801","nonce":2217244566}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":2217244566},{"type":"v1","addr":"172.21.15.163:6803","nonce":2217244566}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":2217244566},{"type":"v1","addr":"172.21.15.163:6807","nonce":2217244566}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":2217244566},{"type":"v1","addr":"172.21.15.163:6805","nonce":2217244566}]},"public_addr":"172.21.15.163:6801/2217244566","cluster_addr":"172.21.15.163:6803/2217244566","heartbeat_back_addr":"172.21.15.163:6807/2217244566","heartbeat_front_addr":"172.21.15.163:6805/2217244566","state":["exists","up"]},{"osd":2,"uuid":"f0ed4131-cd29-4706-89ea-fde40f46ea83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3589694992},{"type":"v1","addr":"172.21.15.187:6801","nonce":3589694992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3589694992},{"type":"v1","addr":"172.21.15.187:6803","nonce":3589694992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3589694992},{"type":"v1","addr":"172.21.15.187:6807","nonce":3589694992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3589694992},{"type":"v1","addr":"172.21.15.187:6805","nonce":3589694992}]},"public_addr":"172.21.15.187:6801/3589694992","cluster_addr":"172.21.15.187:6803/3589694992","heartbeat_back_addr":"172.21.15.187:6807/3589694992","heartbeat_front_addr":"172.21.15.187:6805/3589694992","state":["exists","up"]},{"osd":3,"uuid":"da821310-b502-4b77-a5ce-0ffed0a3dafc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3417210415},{"type":"v1","addr":"172.21.15.29:6811","nonce":3417210415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":3417210415},{"type":"v1","addr":"172.21.15.29:6813","nonce":3417210415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3417210415},{"type":"v1","addr":"172.21.15.29:6817","nonce":3417210415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3417210415},{"type":"v1","addr":"172.21.15.29:6815","nonce":3417210415}]},"public_addr":"172.21.15.29:6811/3417210415","cluster_addr":"172.21.15.29:6813/3417210415","heartbeat_back_addr":"172.21.15.29:6817/3417210415","heartbeat_front_addr":"172.21.15.29:6815/3417210415","state":["exists","up"]},{"osd":4,"uuid":"8b3a348f-e7dd-4ccb-82a5-2d3f5d101da8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3283787183},{"type":"v1","addr":"172.21.15.163:6809","nonce":3283787183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":3283787183},{"type":"v1","addr":"172.21.15.163:6811","nonce":3283787183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3283787183},{"type":"v1","addr":"172.21.15.163:6815","nonce":3283787183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":3283787183},{"type":"v1","addr":"172.21.15.163:6813","nonce":3283787183}]},"public_addr":"172.21.15.163:6809/3283787183","cluster_addr":"172.21.15.163:6811/3283787183","heartbeat_back_addr":"172.21.15.163:6815/3283787183","heartbeat_front_addr":"172.21.15.163:6813/3283787183","state":["exists","up"]},{"osd":5,"uuid":"c26c7155-618b-4712-acdb-d4fd6c8b7f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":2259228817},{"type":"v1","addr":"172.21.15.187:6809","nonce":2259228817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":2259228817},{"type":"v1","addr":"172.21.15.187:6811","nonce":2259228817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":2259228817},{"type":"v1","addr":"172.21.15.187:6815","nonce":2259228817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":2259228817},{"type":"v1","addr":"172.21.15.187:6813","nonce":2259228817}]},"public_addr":"172.21.15.187:6809/2259228817","cluster_addr":"172.21.15.187:6811/2259228817","heartbeat_back_addr":"172.21.15.187:6815/2259228817","heartbeat_front_addr":"172.21.15.187:6813/2259228817","state":["exists","up"]},{"osd":6,"uuid":"042ac0b4-1fe1-444f-bf51-ad89613a2185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":938400701},{"type":"v1","addr":"172.21.15.29:6819","nonce":938400701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":938400701},{"type":"v1","addr":"172.21.15.29:6821","nonce":938400701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":938400701},{"type":"v1","addr":"172.21.15.29:6825","nonce":938400701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":938400701},{"type":"v1","addr":"172.21.15.29:6823","nonce":938400701}]},"public_addr":"172.21.15.29:6819/938400701","cluster_addr":"172.21.15.29:6821/938400701","heartbeat_back_addr":"172.21.15.29:6825/938400701","heartbeat_front_addr":"172.21.15.29:6823/938400701","state":["exists","up"]},{"osd":7,"uuid":"add3d09b-76f3-4a93-83a2-30ac94c3fad1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2783976092},{"type":"v1","addr":"172.21.15.163:6817","nonce":2783976092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2783976092},{"type":"v1","addr":"172.21.15.163:6819","nonce":2783976092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2783976092},{"type":"v1","addr":"172.21.15.163:6823","nonce":2783976092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2783976092},{"type":"v1","addr":"172.21.15.163:6821","nonce":2783976092}]},"public_addr":"172.21.15.163:6817/2783976092","cluster_addr":"172.21.15.163:6819/2783976092","heartbeat_back_addr":"172.21.15.163:6823/2783976092","heartbeat_front_addr":"172.21.15.163:6821/2783976092","state":["exists","up"]},{"osd":8,"uuid":"71044d6a-b55f-4bcf-94c5-79f71eb7b422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":2856803882},{"type":"v1","addr":"172.21.15.187:6817","nonce":2856803882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":2856803882},{"type":"v1","addr":"172.21.15.187:6819","nonce":2856803882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":2856803882},{"type":"v1","addr":"172.21.15.187:6823","nonce":2856803882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":2856803882},{"type":"v1","addr":"172.21.15.187:6821","nonce":2856803882}]},"public_addr":"172.21.15.187:6817/2856803882","cluster_addr":"172.21.15.187:6819/2856803882","heartbeat_back_addr":"172.21.15.187:6823/2856803882","heartbeat_front_addr":"172.21.15.187:6821/2856803882","state":["exists","up"]},{"osd":9,"uuid":"cf417a8e-e576-43de-b89c-f104ee3969ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1392813017},{"type":"v1","addr":"172.21.15.29:6827","nonce":1392813017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1392813017},{"type":"v1","addr":"172.21.15.29:6829","nonce":1392813017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1392813017},{"type":"v1","addr":"172.21.15.29:6833","nonce":1392813017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1392813017},{"type":"v1","addr":"172.21.15.29:6831","nonce":1392813017}]},"public_addr":"172.21.15.29:6827/1392813017","cluster_addr":"172.21.15.29:6829/1392813017","heartbeat_back_addr":"172.21.15.29:6833/1392813017","heartbeat_front_addr":"172.21.15.29:6831/1392813017","state":["exists","up"]},{"osd":10,"uuid":"bc7cfb9d-3bfa-45b2-929a-a8dc90f1a2a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":2523735721},{"type":"v1","addr":"172.21.15.163:6825","nonce":2523735721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":2523735721},{"type":"v1","addr":"172.21.15.163:6827","nonce":2523735721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":2523735721},{"type":"v1","addr":"172.21.15.163:6831","nonce":2523735721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":2523735721},{"type":"v1","addr":"172.21.15.163:6829","nonce":2523735721}]},"public_addr":"172.21.15.163:6825/2523735721","cluster_addr":"172.21.15.163:6827/2523735721","heartbeat_back_addr":"172.21.15.163:6831/2523735721","heartbeat_front_addr":"172.21.15.163:6829/2523735721","state":["exists","up"]},{"osd":11,"uuid":"1453684e-fe11-4cbc-a868-cf90b3a7a021","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1828122314},{"type":"v1","addr":"172.21.15.187:6825","nonce":1828122314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1828122314},{"type":"v1","addr":"172.21.15.187:6827","nonce":1828122314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":1828122314},{"type":"v1","addr":"172.21.15.187:6831","nonce":1828122314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":1828122314},{"type":"v1","addr":"172.21.15.187:6829","nonce":1828122314}]},"public_addr":"172.21.15.187:6825/1828122314","cluster_addr":"172.21.15.187:6827/1828122314","heartbeat_back_addr":"172.21.15.187:6831/1828122314","heartbeat_front_addr":"172.21.15.187:6829/1828122314","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:31:43.572543+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:32:00.759082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:32:17.705002+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:32:35.666788+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:32:53.221701+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:33:10.290988+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:33:27.946743+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:33:45.698074+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:34:02.750761+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:34:20.662278+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:34:38.233389+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:34:55.419540+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/1419474286":"2024-01-25T13:30:30.728638+0000","172.21.15.29:6800/835547951":"2024-01-25T13:30:08.775587+0000","172.21.15.29:6801/1996568496":"2024-01-25T13:30:30.728638+0000","172.21.15.29:0/2727527401":"2024-01-25T13:30:08.775587+0000","172.21.15.29:0/41509684":"2024-01-25T13:30:30.728638+0000","172.21.15.29:6800/1996568496":"2024-01-25T13:30:30.728638+0000","172.21.15.29:0/2833864599":"2024-01-25T13:30:30.728638+0000","172.21.15.29:0/168349482":"2024-01-25T13:30:08.775587+0000","172.21.15.29:6801/835547951":"2024-01-25T13:30:08.775587+0000","172.21.15.29:0/1142561996":"2024-01-25T13:30:08.775587+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T13:34:59.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.228+0000 7f9807caf700 1 -- 172.21.15.29:0/3493522125 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f97ec0a0520 msgr2=0x7f97ec0a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:34:59.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.228+0000 7f9807caf700 1 --2- 172.21.15.29:0/3493522125 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f97ec0a0520 0x7f97ec0a29e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f97f0006010 tx=0x7f97f000b540 comp rx=0 tx=0).stop 2024-01-24T13:34:59.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.228+0000 7f9807caf700 1 -- 172.21.15.29:0/3493522125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9800068860 msgr2=0x7f980019d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:34:59.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.228+0000 7f9807caf700 1 --2- 172.21.15.29:0/3493522125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9800068860 0x7f980019d4c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f97fc00d950 tx=0x7f97fc00dc60 comp rx=0 tx=0).stop 2024-01-24T13:34:59.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.228+0000 7f9807caf700 1 -- 172.21.15.29:0/3493522125 shutdown_connections 2024-01-24T13:34:59.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.228+0000 7f9807caf700 1 --2- 172.21.15.29:0/3493522125 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9800105d00 0x7f98001a1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:59.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.228+0000 7f9807caf700 1 --2- 172.21.15.29:0/3493522125 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f97ec0a0520 0x7f97ec0a29e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:59.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.228+0000 7f9807caf700 1 --2- 172.21.15.29:0/3493522125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9800068860 0x7f980019d4c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:59.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.228+0000 7f9807caf700 1 --2- 172.21.15.29:0/3493522125 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9800103cd0 0x7f980019da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:34:59.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.228+0000 7f9807caf700 1 -- 172.21.15.29:0/3493522125 >> 172.21.15.29:0/3493522125 conn(0x7f9800076d40 msgr2=0x7f98000fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:34:59.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.228+0000 7f9807caf700 1 -- 172.21.15.29:0/3493522125 shutdown_connections 2024-01-24T13:34:59.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:34:59.228+0000 7f9807caf700 1 -- 172.21.15.29:0/3493522125 wait complete. 2024-01-24T13:34:59.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:34:59 smithi163 ceph-mon[44969]: osdmap e77: 12 total, 12 up, 12 in 2024-01-24T13:34:59.551 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:34:59 smithi029 ceph-mon[40651]: osdmap e77: 12 total, 12 up, 12 in 2024-01-24T13:34:59.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:34:59 smithi187 ceph-mon[44911]: osdmap e77: 12 total, 12 up, 12 in 2024-01-24T13:34:59.610 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T13:32:21.264634+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T13:34:59.610 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-01-24T13:34:59.861 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T13:35:00.280 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:00 smithi029 ceph-mon[40651]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:00.280 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:00 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/3493522125' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T13:35:00.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:00 smithi163 ceph-mon[44969]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:00.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:00 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/3493522125' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T13:35:00.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:00 smithi187 ceph-mon[44911]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:00.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:00 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/3493522125' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T13:35:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.815+0000 7f83c862b700 1 -- 172.21.15.29:0/3179494698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83c0100fd0 msgr2=0x7f83c0101450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.815+0000 7f83c862b700 1 --2- 172.21.15.29:0/3179494698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83c0100fd0 0x7f83c0101450 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f83b0009990 tx=0x7f83b0009ca0 comp rx=0 tx=0).stop 2024-01-24T13:35:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.815+0000 7f83c862b700 1 -- 172.21.15.29:0/3179494698 shutdown_connections 2024-01-24T13:35:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.815+0000 7f83c862b700 1 --2- 172.21.15.29:0/3179494698 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83c0101990 0x7f83c010dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.815+0000 7f83c862b700 1 --2- 172.21.15.29:0/3179494698 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f83c0106e80 0x7f83c0107260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.815+0000 7f83c862b700 1 --2- 172.21.15.29:0/3179494698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83c0100fd0 0x7f83c0101450 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.815+0000 7f83c862b700 1 -- 172.21.15.29:0/3179494698 >> 172.21.15.29:0/3179494698 conn(0x7f83c0078d30 msgr2=0x7f83c0079140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:00.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.815+0000 7f83c862b700 1 -- 172.21.15.29:0/3179494698 shutdown_connections 2024-01-24T13:35:00.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.815+0000 7f83c862b700 1 -- 172.21.15.29:0/3179494698 wait complete. 2024-01-24T13:35:00.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.816+0000 7f83c862b700 1 Processor -- start 2024-01-24T13:35:00.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.816+0000 7f83c862b700 1 -- start start 2024-01-24T13:35:00.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.816+0000 7f83c862b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83c0100fd0 0x7f83c0196e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:00.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.816+0000 7f83c862b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f83c0101990 0x7f83c0197340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:00.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.816+0000 7f83c862b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83c0106e80 0x7f83c019b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:00.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.816+0000 7f83c862b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c019bdc0 con 0x7f83c0100fd0 2024-01-24T13:35:00.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.816+0000 7f83c862b700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c019bf30 con 0x7f83c0101990 2024-01-24T13:35:00.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.816+0000 7f83c862b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c019c0f0 con 0x7f83c0106e80 2024-01-24T13:35:00.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.816+0000 7f83c63c7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83c0100fd0 0x7f83c0196e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:00.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.816+0000 7f83c63c7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83c0100fd0 0x7f83c0196e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39864/0 (socket says 172.21.15.29:39864) 2024-01-24T13:35:00.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.816+0000 7f83c63c7700 1 -- 172.21.15.29:0/785141572 learned_addr learned my addr 172.21.15.29:0/785141572 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:00.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.816+0000 7f83c5bc6700 1 --2- 172.21.15.29:0/785141572 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f83c0101990 0x7f83c0197340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:00.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.817+0000 7f83c6bc8700 1 --2- 172.21.15.29:0/785141572 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83c0106e80 0x7f83c019b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:00.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.817+0000 7f83c63c7700 1 -- 172.21.15.29:0/785141572 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83c0106e80 msgr2=0x7f83c019b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:00.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.817+0000 7f83c63c7700 1 --2- 172.21.15.29:0/785141572 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83c0106e80 0x7f83c019b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:00.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.817+0000 7f83c63c7700 1 -- 172.21.15.29:0/785141572 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f83c0101990 msgr2=0x7f83c0197340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:00.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.817+0000 7f83c63c7700 1 --2- 172.21.15.29:0/785141572 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f83c0101990 0x7f83c0197340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:00.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.817+0000 7f83c63c7700 1 -- 172.21.15.29:0/785141572 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b00095f0 con 0x7f83c0100fd0 2024-01-24T13:35:00.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.817+0000 7f83c6bc8700 1 --2- 172.21.15.29:0/785141572 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83c0106e80 0x7f83c019b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T13:35:00.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.817+0000 7f83c63c7700 1 --2- 172.21.15.29:0/785141572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83c0100fd0 0x7f83c0196e00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f83bc00e9a0 tx=0x7f83bc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:00.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.817+0000 7f83b77fe700 1 -- 172.21.15.29:0/785141572 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83bc009790 con 0x7f83c0100fd0 2024-01-24T13:35:00.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.817+0000 7f83b77fe700 1 -- 172.21.15.29:0/785141572 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f83bc00cbd0 con 0x7f83c0100fd0 2024-01-24T13:35:00.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.817+0000 7f83c862b700 1 -- 172.21.15.29:0/785141572 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83c019c3d0 con 0x7f83c0100fd0 2024-01-24T13:35:00.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.817+0000 7f83b77fe700 1 -- 172.21.15.29:0/785141572 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83bc018680 con 0x7f83c0100fd0 2024-01-24T13:35:00.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.817+0000 7f83c862b700 1 -- 172.21.15.29:0/785141572 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83c01a3c30 con 0x7f83c0100fd0 2024-01-24T13:35:00.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.818+0000 7f83c862b700 1 -- 172.21.15.29:0/785141572 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83c004ea40 con 0x7f83c0100fd0 2024-01-24T13:35:00.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.819+0000 7f83b77fe700 1 -- 172.21.15.29:0/785141572 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f83bc00cd40 con 0x7f83c0100fd0 2024-01-24T13:35:00.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.820+0000 7f83b77fe700 1 --2- 172.21.15.29:0/785141572 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f83ac0a0620 0x7f83ac0a2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:00.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.820+0000 7f83c5bc6700 1 --2- 172.21.15.29:0/785141572 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f83ac0a0620 0x7f83ac0a2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:00.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.820+0000 7f83b77fe700 1 -- 172.21.15.29:0/785141572 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f83bc014070 con 0x7f83c0100fd0 2024-01-24T13:35:00.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.821+0000 7f83c5bc6700 1 --2- 172.21.15.29:0/785141572 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f83ac0a0620 0x7f83ac0a2ae0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f83c0198420 tx=0x7f83b0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:00.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:00.825+0000 7f83b77fe700 1 -- 172.21.15.29:0/785141572 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f83bc052120 con 0x7f83c0100fd0 2024-01-24T13:35:01.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:01.032+0000 7f83c862b700 1 -- 172.21.15.29:0/785141572 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f83c0198160 con 0x7f83c0100fd0 2024-01-24T13:35:01.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:01.032+0000 7f83b77fe700 1 -- 172.21.15.29:0/785141572 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7f83c0198160 con 0x7f83c0100fd0 2024-01-24T13:35:01.033 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-01-24T13:35:01.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:01.034+0000 7f83c862b700 1 -- 172.21.15.29:0/785141572 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f83ac0a0620 msgr2=0x7f83ac0a2ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:01.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:01.034+0000 7f83c862b700 1 --2- 172.21.15.29:0/785141572 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f83ac0a0620 0x7f83ac0a2ae0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f83c0198420 tx=0x7f83b0005fd0 comp rx=0 tx=0).stop 2024-01-24T13:35:01.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:01.034+0000 7f83c862b700 1 -- 172.21.15.29:0/785141572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83c0100fd0 msgr2=0x7f83c0196e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:01.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:01.035+0000 7f83c862b700 1 --2- 172.21.15.29:0/785141572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83c0100fd0 0x7f83c0196e00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f83bc00e9a0 tx=0x7f83bc00ecb0 comp rx=0 tx=0).stop 2024-01-24T13:35:01.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:01.035+0000 7f83c862b700 1 -- 172.21.15.29:0/785141572 shutdown_connections 2024-01-24T13:35:01.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:01.035+0000 7f83c862b700 1 --2- 172.21.15.29:0/785141572 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83c0106e80 0x7f83c019b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:01.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:01.035+0000 7f83c862b700 1 --2- 172.21.15.29:0/785141572 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f83ac0a0620 0x7f83ac0a2ae0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:01.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:01.035+0000 7f83c862b700 1 --2- 172.21.15.29:0/785141572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83c0100fd0 0x7f83c0196e00 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:01.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:01.035+0000 7f83c862b700 1 --2- 172.21.15.29:0/785141572 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f83c0101990 0x7f83c0197340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:01.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:01.035+0000 7f83c862b700 1 -- 172.21.15.29:0/785141572 >> 172.21.15.29:0/785141572 conn(0x7f83c0078d30 msgr2=0x7f83c00ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:01.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:01.035+0000 7f83c862b700 1 -- 172.21.15.29:0/785141572 shutdown_connections 2024-01-24T13:35:01.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:01.035+0000 7f83c862b700 1 -- 172.21.15.29:0/785141572 wait complete. 2024-01-24T13:35:01.372 INFO:tasks.cephadm:Adding mds.a on smithi029 2024-01-24T13:35:01.372 INFO:tasks.cephadm:Adding mds.d on smithi029 2024-01-24T13:35:01.373 INFO:tasks.cephadm:Adding mds.g on smithi029 2024-01-24T13:35:01.373 INFO:tasks.cephadm:Adding mds.j on smithi029 2024-01-24T13:35:01.373 INFO:tasks.cephadm:Adding mds.b on smithi163 2024-01-24T13:35:01.373 INFO:tasks.cephadm:Adding mds.e on smithi163 2024-01-24T13:35:01.373 INFO:tasks.cephadm:Adding mds.h on smithi163 2024-01-24T13:35:01.373 INFO:tasks.cephadm:Adding mds.k on smithi163 2024-01-24T13:35:01.373 INFO:tasks.cephadm:Adding mds.c on smithi187 2024-01-24T13:35:01.373 INFO:tasks.cephadm:Adding mds.f on smithi187 2024-01-24T13:35:01.373 INFO:tasks.cephadm:Adding mds.i on smithi187 2024-01-24T13:35:01.373 INFO:tasks.cephadm:Adding mds.l on smithi187 2024-01-24T13:35:01.373 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph orch apply mds all '12;smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi163=b;smithi163=e;smithi163=h;smithi163=k;smithi187=c;smithi187=f;smithi187=i;smithi187=l' 2024-01-24T13:35:01.609 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.c/config 2024-01-24T13:35:02.018 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:01 smithi187 ceph-mon[44911]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:02.018 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:01 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/785141572' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-24T13:35:02.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:01 smithi029 ceph-mon[40651]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:02.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:01 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/785141572' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-24T13:35:02.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:01 smithi163 ceph-mon[44969]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:02.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:01 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/785141572' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-24T13:35:02.521 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.520+0000 7feb6a581700 1 -- 172.21.15.187:0/2149331501 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb64101f80 msgr2=0x7feb64102400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:02.521 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.520+0000 7feb6a581700 1 --2- 172.21.15.187:0/2149331501 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb64101f80 0x7feb64102400 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7feb58009990 tx=0x7feb58009ca0 comp rx=0 tx=0).stop 2024-01-24T13:35:02.521 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.520+0000 7feb6a581700 1 -- 172.21.15.187:0/2149331501 shutdown_connections 2024-01-24T13:35:02.522 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.520+0000 7feb6a581700 1 --2- 172.21.15.187:0/2149331501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb64102940 0x7feb64109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:02.522 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.520+0000 7feb6a581700 1 --2- 172.21.15.187:0/2149331501 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feb64100e20 0x7feb64101240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:02.522 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.520+0000 7feb6a581700 1 --2- 172.21.15.187:0/2149331501 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb64101f80 0x7feb64102400 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:02.522 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.520+0000 7feb6a581700 1 -- 172.21.15.187:0/2149331501 >> 172.21.15.187:0/2149331501 conn(0x7feb640fc3c0 msgr2=0x7feb640fe800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:02.522 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.520+0000 7feb6a581700 1 -- 172.21.15.187:0/2149331501 shutdown_connections 2024-01-24T13:35:02.522 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.520+0000 7feb6a581700 1 -- 172.21.15.187:0/2149331501 wait complete. 2024-01-24T13:35:02.522 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.521+0000 7feb6a581700 1 Processor -- start 2024-01-24T13:35:02.522 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.521+0000 7feb6a581700 1 -- start start 2024-01-24T13:35:02.523 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.521+0000 7feb6a581700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb64100e20 0x7feb641969b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:02.524 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.521+0000 7feb6a581700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feb64101f80 0x7feb64196ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:02.524 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.521+0000 7feb6a581700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb64102940 0x7feb6419df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:02.524 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.521+0000 7feb6a581700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb6419e540 con 0x7feb64102940 2024-01-24T13:35:02.524 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.521+0000 7feb6a581700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb6419e680 con 0x7feb64101f80 2024-01-24T13:35:02.524 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.521+0000 7feb6a581700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb6419e810 con 0x7feb64100e20 2024-01-24T13:35:02.524 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.521+0000 7feb63fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb64100e20 0x7feb641969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:02.524 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.521+0000 7feb63fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb64100e20 0x7feb641969b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:57310/0 (socket says 172.21.15.187:57310) 2024-01-24T13:35:02.524 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.521+0000 7feb63fff700 1 -- 172.21.15.187:0/59704612 learned_addr learned my addr 172.21.15.187:0/59704612 (peer_addr_for_me v2:172.21.15.187:0/0) 2024-01-24T13:35:02.524 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.522+0000 7feb637fe700 1 --2- 172.21.15.187:0/59704612 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feb64101f80 0x7feb64196ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:02.524 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.522+0000 7feb68b1e700 1 --2- 172.21.15.187:0/59704612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb64102940 0x7feb6419df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:02.524 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.522+0000 7feb63fff700 1 -- 172.21.15.187:0/59704612 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feb64101f80 msgr2=0x7feb64196ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:02.524 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.522+0000 7feb63fff700 1 --2- 172.21.15.187:0/59704612 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feb64101f80 0x7feb64196ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:02.525 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.522+0000 7feb63fff700 1 -- 172.21.15.187:0/59704612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb64102940 msgr2=0x7feb6419df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:02.525 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.522+0000 7feb63fff700 1 --2- 172.21.15.187:0/59704612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb64102940 0x7feb6419df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:02.525 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.522+0000 7feb63fff700 1 -- 172.21.15.187:0/59704612 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb580095f0 con 0x7feb64100e20 2024-01-24T13:35:02.525 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.522+0000 7feb63fff700 1 --2- 172.21.15.187:0/59704612 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb64100e20 0x7feb641969b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7feb5400d980 tx=0x7feb5400dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:02.525 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.522+0000 7feb617fa700 1 -- 172.21.15.187:0/59704612 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb54009b40 con 0x7feb64100e20 2024-01-24T13:35:02.525 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.522+0000 7feb617fa700 1 -- 172.21.15.187:0/59704612 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb5400b6e0 con 0x7feb64100e20 2024-01-24T13:35:02.525 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.522+0000 7feb6a581700 1 -- 172.21.15.187:0/59704612 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb6419eaf0 con 0x7feb64100e20 2024-01-24T13:35:02.525 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.522+0000 7feb617fa700 1 -- 172.21.15.187:0/59704612 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb5400f620 con 0x7feb64100e20 2024-01-24T13:35:02.525 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.522+0000 7feb6a581700 1 -- 172.21.15.187:0/59704612 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb6419eff0 con 0x7feb64100e20 2024-01-24T13:35:02.525 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.523+0000 7feb6a581700 1 -- 172.21.15.187:0/59704612 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb64108370 con 0x7feb64100e20 2024-01-24T13:35:02.526 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.524+0000 7feb617fa700 1 -- 172.21.15.187:0/59704612 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7feb54010460 con 0x7feb64100e20 2024-01-24T13:35:02.526 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.525+0000 7feb617fa700 1 --2- 172.21.15.187:0/59704612 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7feb4c0a0520 0x7feb4c0a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:02.526 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.525+0000 7feb617fa700 1 -- 172.21.15.187:0/59704612 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7feb540cb760 con 0x7feb64100e20 2024-01-24T13:35:02.526 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.525+0000 7feb637fe700 1 --2- 172.21.15.187:0/59704612 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7feb4c0a0520 0x7feb4c0a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:02.526 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.525+0000 7feb637fe700 1 --2- 172.21.15.187:0/59704612 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7feb4c0a0520 0x7feb4c0a29e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7feb5800b5c0 tx=0x7feb58005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:02.530 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.529+0000 7feb617fa700 1 -- 172.21.15.187:0/59704612 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7feb540957a0 con 0x7feb64100e20 2024-01-24T13:35:02.727 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.726+0000 7feb6a581700 1 -- 172.21.15.187:0/59704612 --> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi163=b;smithi163=e;smithi163=h;smithi163=k;smithi187=c;smithi187=f;smithi187=i;smithi187=l", "target": ["mon-mgr", ""]}) v1 -- 0x7feb64061280 con 0x7feb4c0a0520 2024-01-24T13:35:02.736 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.735+0000 7feb617fa700 1 -- 172.21.15.187:0/59704612 <== mgr.14152 v2:172.21.15.29:6800/2498991338 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7feb64061280 con 0x7feb4c0a0520 2024-01-24T13:35:02.736 INFO:teuthology.orchestra.run.smithi187.stdout:Scheduled mds.all update... 2024-01-24T13:35:02.738 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.737+0000 7feb6a581700 1 -- 172.21.15.187:0/59704612 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7feb4c0a0520 msgr2=0x7feb4c0a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:02.738 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.737+0000 7feb6a581700 1 --2- 172.21.15.187:0/59704612 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7feb4c0a0520 0x7feb4c0a29e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7feb5800b5c0 tx=0x7feb58005960 comp rx=0 tx=0).stop 2024-01-24T13:35:02.738 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.737+0000 7feb6a581700 1 -- 172.21.15.187:0/59704612 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb64100e20 msgr2=0x7feb641969b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:02.738 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.737+0000 7feb6a581700 1 --2- 172.21.15.187:0/59704612 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb64100e20 0x7feb641969b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7feb5400d980 tx=0x7feb5400dc90 comp rx=0 tx=0).stop 2024-01-24T13:35:02.738 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.738+0000 7feb6a581700 1 -- 172.21.15.187:0/59704612 shutdown_connections 2024-01-24T13:35:02.738 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.738+0000 7feb6a581700 1 --2- 172.21.15.187:0/59704612 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7feb4c0a0520 0x7feb4c0a29e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:02.739 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.738+0000 7feb6a581700 1 --2- 172.21.15.187:0/59704612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb64102940 0x7feb6419df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:02.739 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.738+0000 7feb6a581700 1 --2- 172.21.15.187:0/59704612 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb64100e20 0x7feb641969b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:02.739 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.738+0000 7feb6a581700 1 --2- 172.21.15.187:0/59704612 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feb64101f80 0x7feb64196ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:02.739 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.738+0000 7feb6a581700 1 -- 172.21.15.187:0/59704612 >> 172.21.15.187:0/59704612 conn(0x7feb640fc3c0 msgr2=0x7feb641054b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:02.739 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.738+0000 7feb6a581700 1 -- 172.21.15.187:0/59704612 shutdown_connections 2024-01-24T13:35:02.739 INFO:teuthology.orchestra.run.smithi187.stderr:2024-01-24T13:35:02.738+0000 7feb6a581700 1 -- 172.21.15.187:0/59704612 wait complete. 2024-01-24T13:35:03.097 DEBUG:teuthology.orchestra.run.smithi029:mds.a> sudo journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mds.a.service 2024-01-24T13:35:03.100 DEBUG:teuthology.orchestra.run.smithi029:mds.d> sudo journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mds.d.service 2024-01-24T13:35:03.134 INFO:journalctl@ceph.mds.a.smithi029.stdout:-- Logs begin at Wed 2024-01-24 13:13:42 UTC. -- 2024-01-24T13:35:03.135 DEBUG:teuthology.orchestra.run.smithi029:mds.g> sudo journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mds.g.service 2024-01-24T13:35:03.167 INFO:journalctl@ceph.mds.d.smithi029.stdout:-- Logs begin at Wed 2024-01-24 13:13:42 UTC. -- 2024-01-24T13:35:03.172 DEBUG:teuthology.orchestra.run.smithi029:mds.j> sudo journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mds.j.service 2024-01-24T13:35:03.198 INFO:journalctl@ceph.mds.g.smithi029.stdout:-- Logs begin at Wed 2024-01-24 13:13:42 UTC. -- 2024-01-24T13:35:03.199 DEBUG:teuthology.orchestra.run.smithi163:mds.b> sudo journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mds.b.service 2024-01-24T13:35:03.201 DEBUG:teuthology.orchestra.run.smithi163:mds.e> sudo journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mds.e.service 2024-01-24T13:35:03.229 INFO:journalctl@ceph.mds.j.smithi029.stdout:-- Logs begin at Wed 2024-01-24 13:13:42 UTC. -- 2024-01-24T13:35:03.239 INFO:journalctl@ceph.mds.b.smithi163.stdout:-- Logs begin at Wed 2024-01-24 13:14:09 UTC. -- 2024-01-24T13:35:03.241 DEBUG:teuthology.orchestra.run.smithi163:mds.h> sudo journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mds.h.service 2024-01-24T13:35:03.270 INFO:journalctl@ceph.mds.e.smithi163.stdout:-- Logs begin at Wed 2024-01-24 13:14:09 UTC. -- 2024-01-24T13:35:03.271 DEBUG:teuthology.orchestra.run.smithi163:mds.k> sudo journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mds.k.service 2024-01-24T13:35:03.299 INFO:journalctl@ceph.mds.h.smithi163.stdout:-- Logs begin at Wed 2024-01-24 13:14:09 UTC. -- 2024-01-24T13:35:03.300 DEBUG:teuthology.orchestra.run.smithi187:mds.c> sudo journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mds.c.service 2024-01-24T13:35:03.302 DEBUG:teuthology.orchestra.run.smithi187:mds.f> sudo journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mds.f.service 2024-01-24T13:35:03.327 INFO:journalctl@ceph.mds.k.smithi163.stdout:-- Logs begin at Wed 2024-01-24 13:14:09 UTC. -- 2024-01-24T13:35:03.331 INFO:journalctl@ceph.mds.c.smithi187.stdout:-- Logs begin at Wed 2024-01-24 13:13:23 UTC. -- 2024-01-24T13:35:03.332 DEBUG:teuthology.orchestra.run.smithi187:mds.i> sudo journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mds.i.service 2024-01-24T13:35:03.359 INFO:journalctl@ceph.mds.f.smithi187.stdout:-- Logs begin at Wed 2024-01-24 13:13:23 UTC. -- 2024-01-24T13:35:03.360 DEBUG:teuthology.orchestra.run.smithi187:mds.l> sudo journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mds.l.service 2024-01-24T13:35:03.385 INFO:journalctl@ceph.mds.i.smithi187.stdout:-- Logs begin at Wed 2024-01-24 13:13:23 UTC. -- 2024-01-24T13:35:03.386 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-01-24T13:35:03.386 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T13:35:03.414 INFO:journalctl@ceph.mds.l.smithi187.stdout:-- Logs begin at Wed 2024-01-24 13:13:23 UTC. -- 2024-01-24T13:35:03.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.543+0000 7f7331edb700 1 -- 172.21.15.29:0/2345896038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f732c1017a0 msgr2=0x7f732c103b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:03.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.543+0000 7f7331edb700 1 --2- 172.21.15.29:0/2345896038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f732c1017a0 0x7f732c103b90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f731c009990 tx=0x7f731c009ca0 comp rx=0 tx=0).stop 2024-01-24T13:35:03.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.543+0000 7f7331edb700 1 -- 172.21.15.29:0/2345896038 shutdown_connections 2024-01-24T13:35:03.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.543+0000 7f7331edb700 1 --2- 172.21.15.29:0/2345896038 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f732c106a00 0x7f732c108e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:03.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.543+0000 7f7331edb700 1 --2- 172.21.15.29:0/2345896038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f732c1017a0 0x7f732c103b90 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:03.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.543+0000 7f7331edb700 1 --2- 172.21.15.29:0/2345896038 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f732c1040d0 0x7f732c1064c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:03.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.543+0000 7f7331edb700 1 -- 172.21.15.29:0/2345896038 >> 172.21.15.29:0/2345896038 conn(0x7f732c0fb170 msgr2=0x7f732c0fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:03.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.543+0000 7f7331edb700 1 -- 172.21.15.29:0/2345896038 shutdown_connections 2024-01-24T13:35:03.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.543+0000 7f7331edb700 1 -- 172.21.15.29:0/2345896038 wait complete. 2024-01-24T13:35:03.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.544+0000 7f7331edb700 1 Processor -- start 2024-01-24T13:35:03.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.544+0000 7f7331edb700 1 -- start start 2024-01-24T13:35:03.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.545+0000 7f7331edb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f732c1017a0 0x7f732c196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:03.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.545+0000 7f7331edb700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f732c1040d0 0x7f732c196f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:03.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.545+0000 7f7331edb700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f732c106a00 0x7f732c19dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:03.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.545+0000 7f7331edb700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732c19e590 con 0x7f732c1017a0 2024-01-24T13:35:03.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.545+0000 7f7331edb700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732c19e6d0 con 0x7f732c1040d0 2024-01-24T13:35:03.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.545+0000 7f7331edb700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732c19e860 con 0x7f732c106a00 2024-01-24T13:35:03.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.545+0000 7f732b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f732c1017a0 0x7f732c196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:03.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.545+0000 7f732b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f732c1017a0 0x7f732c196a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55846/0 (socket says 172.21.15.29:55846) 2024-01-24T13:35:03.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.545+0000 7f732b7fe700 1 -- 172.21.15.29:0/2256076327 learned_addr learned my addr 172.21.15.29:0/2256076327 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:03.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.545+0000 7f732affd700 1 --2- 172.21.15.29:0/2256076327 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f732c1040d0 0x7f732c196f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:03.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.545+0000 7f732bfff700 1 --2- 172.21.15.29:0/2256076327 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f732c106a00 0x7f732c19dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:03.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.545+0000 7f732b7fe700 1 -- 172.21.15.29:0/2256076327 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f732c106a00 msgr2=0x7f732c19dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:03.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.545+0000 7f732b7fe700 1 --2- 172.21.15.29:0/2256076327 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f732c106a00 0x7f732c19dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:03.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.545+0000 7f732b7fe700 1 -- 172.21.15.29:0/2256076327 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f732c1040d0 msgr2=0x7f732c196f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:03.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.545+0000 7f732b7fe700 1 --2- 172.21.15.29:0/2256076327 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f732c1040d0 0x7f732c196f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:03.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.545+0000 7f732b7fe700 1 -- 172.21.15.29:0/2256076327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f731c0095f0 con 0x7f732c1017a0 2024-01-24T13:35:03.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.545+0000 7f732bfff700 1 --2- 172.21.15.29:0/2256076327 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f732c106a00 0x7f732c19dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T13:35:03.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.545+0000 7f732b7fe700 1 --2- 172.21.15.29:0/2256076327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f732c1017a0 0x7f732c196a10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f731c0059d0 tx=0x7f731c00f510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:03.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.545+0000 7f7328ff9700 1 -- 172.21.15.29:0/2256076327 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f731c01d070 con 0x7f732c1017a0 2024-01-24T13:35:03.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.545+0000 7f7331edb700 1 -- 172.21.15.29:0/2256076327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f732c19eae0 con 0x7f732c1017a0 2024-01-24T13:35:03.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.545+0000 7f7328ff9700 1 -- 172.21.15.29:0/2256076327 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f731c00fd60 con 0x7f732c1017a0 2024-01-24T13:35:03.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.545+0000 7f7328ff9700 1 -- 172.21.15.29:0/2256076327 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f731c017650 con 0x7f732c1017a0 2024-01-24T13:35:03.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.546+0000 7f7331edb700 1 -- 172.21.15.29:0/2256076327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f732c19ef80 con 0x7f732c1017a0 2024-01-24T13:35:03.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.546+0000 7f7331edb700 1 -- 172.21.15.29:0/2256076327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f732c0fccf0 con 0x7f732c1017a0 2024-01-24T13:35:03.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.548+0000 7f7328ff9700 1 -- 172.21.15.29:0/2256076327 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f731c0177b0 con 0x7f732c1017a0 2024-01-24T13:35:03.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.548+0000 7f7328ff9700 1 --2- 172.21.15.29:0/2256076327 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f73140a0570 0x7f73140a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:03.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.548+0000 7f732affd700 1 --2- 172.21.15.29:0/2256076327 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f73140a0570 0x7f73140a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:03.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.548+0000 7f7328ff9700 1 -- 172.21.15.29:0/2256076327 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f731c0cca90 con 0x7f732c1017a0 2024-01-24T13:35:03.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.549+0000 7f732affd700 1 --2- 172.21.15.29:0/2256076327 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f73140a0570 0x7f73140a2a30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7320009bf0 tx=0x7f7320009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:03.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.553+0000 7f7328ff9700 1 -- 172.21.15.29:0/2256076327 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f731c05d120 con 0x7f732c1017a0 2024-01-24T13:35:03.739 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:03 smithi029 ceph-mon[40651]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi163=b;smithi163=e;smithi163=h;smithi163=k;smithi187=c;smithi187=f;smithi187=i;smithi187=l", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:35:03.739 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:03 smithi029 ceph-mon[40651]: Saving service mds.all spec with placement smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi163=b;smithi163=e;smithi163=h;smithi163=k;smithi187=c;smithi187=f;smithi187=i;smithi187=l;count:12 2024-01-24T13:35:03.739 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:03 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:03.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.770+0000 7f7331edb700 1 -- 172.21.15.29:0/2256076327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f732c02cec0 con 0x7f732c1017a0 2024-01-24T13:35:03.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.771+0000 7f7328ff9700 1 -- 172.21.15.29:0/2256076327 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+18443 (secure 0 0 0) 0x7f731c096b60 con 0x7f732c1017a0 2024-01-24T13:35:03.771 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:35:03.771 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":77,"fsid":"83afcf5e-babc-11ee-95b1-87774f69a715","created":"2024-01-24T13:29:33.175895+0000","modified":"2024-01-24T13:34:58.148429+0000","last_up_change":"2024-01-24T13:34:57.141902+0000","last_in_change":"2024-01-24T13:34:41.071943+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T13:32:21.264634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9857a607-2e6f-41a5-a86a-795970be6210","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":2406629076},{"type":"v1","addr":"172.21.15.29:6803","nonce":2406629076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":2406629076},{"type":"v1","addr":"172.21.15.29:6805","nonce":2406629076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2406629076},{"type":"v1","addr":"172.21.15.29:6809","nonce":2406629076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":2406629076},{"type":"v1","addr":"172.21.15.29:6807","nonce":2406629076}]},"public_addr":"172.21.15.29:6803/2406629076","cluster_addr":"172.21.15.29:6805/2406629076","heartbeat_back_addr":"172.21.15.29:6809/2406629076","heartbeat_front_addr":"172.21.15.29:6807/2406629076","state":["exists","up"]},{"osd":1,"uuid":"8bb2ba48-23d4-4bbd-a912-ae74f9e7f6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":2217244566},{"type":"v1","addr":"172.21.15.163:6801","nonce":2217244566}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":2217244566},{"type":"v1","addr":"172.21.15.163:6803","nonce":2217244566}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":2217244566},{"type":"v1","addr":"172.21.15.163:6807","nonce":2217244566}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":2217244566},{"type":"v1","addr":"172.21.15.163:6805","nonce":2217244566}]},"public_addr":"172.21.15.163:6801/2217244566","cluster_addr":"172.21.15.163:6803/2217244566","heartbeat_back_addr":"172.21.15.163:6807/2217244566","heartbeat_front_addr":"172.21.15.163:6805/2217244566","state":["exists","up"]},{"osd":2,"uuid":"f0ed4131-cd29-4706-89ea-fde40f46ea83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3589694992},{"type":"v1","addr":"172.21.15.187:6801","nonce":3589694992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3589694992},{"type":"v1","addr":"172.21.15.187:6803","nonce":3589694992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3589694992},{"type":"v1","addr":"172.21.15.187:6807","nonce":3589694992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3589694992},{"type":"v1","addr":"172.21.15.187:6805","nonce":3589694992}]},"public_addr":"172.21.15.187:6801/3589694992","cluster_addr":"172.21.15.187:6803/3589694992","heartbeat_back_addr":"172.21.15.187:6807/3589694992","heartbeat_front_addr":"172.21.15.187:6805/3589694992","state":["exists","up"]},{"osd":3,"uuid":"da821310-b502-4b77-a5ce-0ffed0a3dafc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3417210415},{"type":"v1","addr":"172.21.15.29:6811","nonce":3417210415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":3417210415},{"type":"v1","addr":"172.21.15.29:6813","nonce":3417210415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3417210415},{"type":"v1","addr":"172.21.15.29:6817","nonce":3417210415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3417210415},{"type":"v1","addr":"172.21.15.29:6815","nonce":3417210415}]},"public_addr":"172.21.15.29:6811/3417210415","cluster_addr":"172.21.15.29:6813/3417210415","heartbeat_back_addr":"172.21.15.29:6817/3417210415","heartbeat_front_addr":"172.21.15.29:6815/3417210415","state":["exists","up"]},{"osd":4,"uuid":"8b3a348f-e7dd-4ccb-82a5-2d3f5d101da8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3283787183},{"type":"v1","addr":"172.21.15.163:6809","nonce":3283787183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":3283787183},{"type":"v1","addr":"172.21.15.163:6811","nonce":3283787183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3283787183},{"type":"v1","addr":"172.21.15.163:6815","nonce":3283787183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":3283787183},{"type":"v1","addr":"172.21.15.163:6813","nonce":3283787183}]},"public_addr":"172.21.15.163:6809/3283787183","cluster_addr":"172.21.15.163:6811/3283787183","heartbeat_back_addr":"172.21.15.163:6815/3283787183","heartbeat_front_addr":"172.21.15.163:6813/3283787183","state":["exists","up"]},{"osd":5,"uuid":"c26c7155-618b-4712-acdb-d4fd6c8b7f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":2259228817},{"type":"v1","addr":"172.21.15.187:6809","nonce":2259228817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":2259228817},{"type":"v1","addr":"172.21.15.187:6811","nonce":2259228817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":2259228817},{"type":"v1","addr":"172.21.15.187:6815","nonce":2259228817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":2259228817},{"type":"v1","addr":"172.21.15.187:6813","nonce":2259228817}]},"public_addr":"172.21.15.187:6809/2259228817","cluster_addr":"172.21.15.187:6811/2259228817","heartbeat_back_addr":"172.21.15.187:6815/2259228817","heartbeat_front_addr":"172.21.15.187:6813/2259228817","state":["exists","up"]},{"osd":6,"uuid":"042ac0b4-1fe1-444f-bf51-ad89613a2185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":938400701},{"type":"v1","addr":"172.21.15.29:6819","nonce":938400701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":938400701},{"type":"v1","addr":"172.21.15.29:6821","nonce":938400701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":938400701},{"type":"v1","addr":"172.21.15.29:6825","nonce":938400701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":938400701},{"type":"v1","addr":"172.21.15.29:6823","nonce":938400701}]},"public_addr":"172.21.15.29:6819/938400701","cluster_addr":"172.21.15.29:6821/938400701","heartbeat_back_addr":"172.21.15.29:6825/938400701","heartbeat_front_addr":"172.21.15.29:6823/938400701","state":["exists","up"]},{"osd":7,"uuid":"add3d09b-76f3-4a93-83a2-30ac94c3fad1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2783976092},{"type":"v1","addr":"172.21.15.163:6817","nonce":2783976092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2783976092},{"type":"v1","addr":"172.21.15.163:6819","nonce":2783976092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2783976092},{"type":"v1","addr":"172.21.15.163:6823","nonce":2783976092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2783976092},{"type":"v1","addr":"172.21.15.163:6821","nonce":2783976092}]},"public_addr":"172.21.15.163:6817/2783976092","cluster_addr":"172.21.15.163:6819/2783976092","heartbeat_back_addr":"172.21.15.163:6823/2783976092","heartbeat_front_addr":"172.21.15.163:6821/2783976092","state":["exists","up"]},{"osd":8,"uuid":"71044d6a-b55f-4bcf-94c5-79f71eb7b422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":2856803882},{"type":"v1","addr":"172.21.15.187:6817","nonce":2856803882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":2856803882},{"type":"v1","addr":"172.21.15.187:6819","nonce":2856803882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":2856803882},{"type":"v1","addr":"172.21.15.187:6823","nonce":2856803882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":2856803882},{"type":"v1","addr":"172.21.15.187:6821","nonce":2856803882}]},"public_addr":"172.21.15.187:6817/2856803882","cluster_addr":"172.21.15.187:6819/2856803882","heartbeat_back_addr":"172.21.15.187:6823/2856803882","heartbeat_front_addr":"172.21.15.187:6821/2856803882","state":["exists","up"]},{"osd":9,"uuid":"cf417a8e-e576-43de-b89c-f104ee3969ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1392813017},{"type":"v1","addr":"172.21.15.29:6827","nonce":1392813017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1392813017},{"type":"v1","addr":"172.21.15.29:6829","nonce":1392813017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1392813017},{"type":"v1","addr":"172.21.15.29:6833","nonce":1392813017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1392813017},{"type":"v1","addr":"172.21.15.29:6831","nonce":1392813017}]},"public_addr":"172.21.15.29:6827/1392813017","cluster_addr":"172.21.15.29:6829/1392813017","heartbeat_back_addr":"172.21.15.29:6833/1392813017","heartbeat_front_addr":"172.21.15.29:6831/1392813017","state":["exists","up"]},{"osd":10,"uuid":"bc7cfb9d-3bfa-45b2-929a-a8dc90f1a2a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":2523735721},{"type":"v1","addr":"172.21.15.163:6825","nonce":2523735721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":2523735721},{"type":"v1","addr":"172.21.15.163:6827","nonce":2523735721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":2523735721},{"type":"v1","addr":"172.21.15.163:6831","nonce":2523735721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":2523735721},{"type":"v1","addr":"172.21.15.163:6829","nonce":2523735721}]},"public_addr":"172.21.15.163:6825/2523735721","cluster_addr":"172.21.15.163:6827/2523735721","heartbeat_back_addr":"172.21.15.163:6831/2523735721","heartbeat_front_addr":"172.21.15.163:6829/2523735721","state":["exists","up"]},{"osd":11,"uuid":"1453684e-fe11-4cbc-a868-cf90b3a7a021","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1828122314},{"type":"v1","addr":"172.21.15.187:6825","nonce":1828122314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1828122314},{"type":"v1","addr":"172.21.15.187:6827","nonce":1828122314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":1828122314},{"type":"v1","addr":"172.21.15.187:6831","nonce":1828122314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":1828122314},{"type":"v1","addr":"172.21.15.187:6829","nonce":1828122314}]},"public_addr":"172.21.15.187:6825/1828122314","cluster_addr":"172.21.15.187:6827/1828122314","heartbeat_back_addr":"172.21.15.187:6831/1828122314","heartbeat_front_addr":"172.21.15.187:6829/1828122314","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:31:43.572543+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:32:00.759082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:32:17.705002+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:32:35.666788+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:32:53.221701+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:33:10.290988+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:33:27.946743+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:33:45.698074+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:34:02.750761+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:34:20.662278+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:34:38.233389+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:34:55.419540+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/1419474286":"2024-01-25T13:30:30.728638+0000","172.21.15.29:6800/835547951":"2024-01-25T13:30:08.775587+0000","172.21.15.29:6801/1996568496":"2024-01-25T13:30:30.728638+0000","172.21.15.29:0/2727527401":"2024-01-25T13:30:08.775587+0000","172.21.15.29:0/41509684":"2024-01-25T13:30:30.728638+0000","172.21.15.29:6800/1996568496":"2024-01-25T13:30:30.728638+0000","172.21.15.29:0/2833864599":"2024-01-25T13:30:30.728638+0000","172.21.15.29:0/168349482":"2024-01-25T13:30:08.775587+0000","172.21.15.29:6801/835547951":"2024-01-25T13:30:08.775587+0000","172.21.15.29:0/1142561996":"2024-01-25T13:30:08.775587+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T13:35:03.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.773+0000 7f7331edb700 1 -- 172.21.15.29:0/2256076327 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f73140a0570 msgr2=0x7f73140a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:03.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.773+0000 7f7331edb700 1 --2- 172.21.15.29:0/2256076327 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f73140a0570 0x7f73140a2a30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7320009bf0 tx=0x7f7320009260 comp rx=0 tx=0).stop 2024-01-24T13:35:03.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.773+0000 7f7331edb700 1 -- 172.21.15.29:0/2256076327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f732c1017a0 msgr2=0x7f732c196a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:03.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.773+0000 7f7331edb700 1 --2- 172.21.15.29:0/2256076327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f732c1017a0 0x7f732c196a10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f731c0059d0 tx=0x7f731c00f510 comp rx=0 tx=0).stop 2024-01-24T13:35:03.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.773+0000 7f7331edb700 1 -- 172.21.15.29:0/2256076327 shutdown_connections 2024-01-24T13:35:03.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.773+0000 7f7331edb700 1 --2- 172.21.15.29:0/2256076327 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f732c106a00 0x7f732c19dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:03.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.773+0000 7f7331edb700 1 --2- 172.21.15.29:0/2256076327 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f73140a0570 0x7f73140a2a30 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:03.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.773+0000 7f7331edb700 1 --2- 172.21.15.29:0/2256076327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f732c1017a0 0x7f732c196a10 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:03.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.773+0000 7f7331edb700 1 --2- 172.21.15.29:0/2256076327 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f732c1040d0 0x7f732c196f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:03.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.773+0000 7f7331edb700 1 -- 172.21.15.29:0/2256076327 >> 172.21.15.29:0/2256076327 conn(0x7f732c0fb170 msgr2=0x7f732c102410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:03.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.773+0000 7f7331edb700 1 -- 172.21.15.29:0/2256076327 shutdown_connections 2024-01-24T13:35:03.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.774+0000 7f7331edb700 1 -- 172.21.15.29:0/2256076327 wait complete. 2024-01-24T13:35:03.788 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T13:32:21.264634+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T13:35:03.788 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T13:35:03.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.928+0000 7f8f9cfd3700 1 -- 172.21.15.29:0/4150594220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f98100e20 msgr2=0x7f8f98101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:03.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.928+0000 7f8f9cfd3700 1 --2- 172.21.15.29:0/4150594220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f98100e20 0x7f8f98101280 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c0099c0 tx=0x7f8f8c009cd0 comp rx=0 tx=0).stop 2024-01-24T13:35:03.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.928+0000 7f8f9cfd3700 1 -- 172.21.15.29:0/4150594220 shutdown_connections 2024-01-24T13:35:03.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.928+0000 7f8f9cfd3700 1 --2- 172.21.15.29:0/4150594220 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f981017c0 0x7f8f98108070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:03.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.928+0000 7f8f9cfd3700 1 --2- 172.21.15.29:0/4150594220 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8f980ffc30 0x7f8f98100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:03.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.928+0000 7f8f9cfd3700 1 --2- 172.21.15.29:0/4150594220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f98100e20 0x7f8f98101280 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:03.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.928+0000 7f8f9cfd3700 1 -- 172.21.15.29:0/4150594220 >> 172.21.15.29:0/4150594220 conn(0x7f8f980fb1a0 msgr2=0x7f8f980fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:03.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.928+0000 7f8f9cfd3700 1 -- 172.21.15.29:0/4150594220 shutdown_connections 2024-01-24T13:35:03.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.928+0000 7f8f9cfd3700 1 -- 172.21.15.29:0/4150594220 wait complete. 2024-01-24T13:35:03.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.929+0000 7f8f9cfd3700 1 Processor -- start 2024-01-24T13:35:03.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.929+0000 7f8f9cfd3700 1 -- start start 2024-01-24T13:35:03.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.929+0000 7f8f9cfd3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f980ffc30 0x7f8f98195920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:03.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.929+0000 7f8f9cfd3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8f98100e20 0x7f8f98195e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:03.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.929+0000 7f8f9cfd3700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f981017c0 0x7f8f9819ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:03.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.929+0000 7f8f9cfd3700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9819d4a0 con 0x7f8f980ffc30 2024-01-24T13:35:03.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.929+0000 7f8f9cfd3700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9819d5e0 con 0x7f8f981017c0 2024-01-24T13:35:03.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.929+0000 7f8f9cfd3700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9819d770 con 0x7f8f98100e20 2024-01-24T13:35:03.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.929+0000 7f8f9659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f980ffc30 0x7f8f98195920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:03.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.929+0000 7f8f95d9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8f98100e20 0x7f8f98195e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:03.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.929+0000 7f8f9659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f980ffc30 0x7f8f98195920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55876/0 (socket says 172.21.15.29:55876) 2024-01-24T13:35:03.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.930+0000 7f8f9659c700 1 -- 172.21.15.29:0/3515702740 learned_addr learned my addr 172.21.15.29:0/3515702740 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:03.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.930+0000 7f8f96d9d700 1 --2- 172.21.15.29:0/3515702740 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f981017c0 0x7f8f9819ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:03.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.930+0000 7f8f95d9b700 1 -- 172.21.15.29:0/3515702740 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f981017c0 msgr2=0x7f8f9819ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:03.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.930+0000 7f8f95d9b700 1 --2- 172.21.15.29:0/3515702740 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f981017c0 0x7f8f9819ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:03.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.930+0000 7f8f95d9b700 1 -- 172.21.15.29:0/3515702740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f980ffc30 msgr2=0x7f8f98195920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:03.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.930+0000 7f8f95d9b700 1 --2- 172.21.15.29:0/3515702740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f980ffc30 0x7f8f98195920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:03.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.930+0000 7f8f95d9b700 1 -- 172.21.15.29:0/3515702740 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f8c0095f0 con 0x7f8f98100e20 2024-01-24T13:35:03.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.930+0000 7f8f9659c700 1 --2- 172.21.15.29:0/3515702740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f980ffc30 0x7f8f98195920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T13:35:03.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.930+0000 7f8f95d9b700 1 --2- 172.21.15.29:0/3515702740 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8f98100e20 0x7f8f98195e60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c009fd0 tx=0x7f8f8c00f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:03.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.930+0000 7f8f877fe700 1 -- 172.21.15.29:0/3515702740 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f8c01d070 con 0x7f8f98100e20 2024-01-24T13:35:03.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.930+0000 7f8f9cfd3700 1 -- 172.21.15.29:0/3515702740 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f9819d9f0 con 0x7f8f98100e20 2024-01-24T13:35:03.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.930+0000 7f8f877fe700 1 -- 172.21.15.29:0/3515702740 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f8c0054e0 con 0x7f8f98100e20 2024-01-24T13:35:03.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.930+0000 7f8f877fe700 1 -- 172.21.15.29:0/3515702740 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f8c004b00 con 0x7f8f98100e20 2024-01-24T13:35:03.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.930+0000 7f8f9cfd3700 1 -- 172.21.15.29:0/3515702740 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f9819ddc0 con 0x7f8f98100e20 2024-01-24T13:35:03.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.931+0000 7f8f9cfd3700 1 -- 172.21.15.29:0/3515702740 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f98106bb0 con 0x7f8f98100e20 2024-01-24T13:35:03.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.932+0000 7f8f877fe700 1 -- 172.21.15.29:0/3515702740 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f8f8c00fb80 con 0x7f8f98100e20 2024-01-24T13:35:03.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.933+0000 7f8f877fe700 1 --2- 172.21.15.29:0/3515702740 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f8f800a0520 0x7f8f800a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:03.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.933+0000 7f8f9659c700 1 --2- 172.21.15.29:0/3515702740 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f8f800a0520 0x7f8f800a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:03.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.933+0000 7f8f877fe700 1 -- 172.21.15.29:0/3515702740 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f8f8c0cc1b0 con 0x7f8f98100e20 2024-01-24T13:35:03.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.934+0000 7f8f9659c700 1 --2- 172.21.15.29:0/3515702740 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f8f800a0520 0x7f8f800a29e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8f88005950 tx=0x7f8f88009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:03.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:03.936+0000 7f8f877fe700 1 -- 172.21.15.29:0/3515702740 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f8f8c096270 con 0x7f8f98100e20 2024-01-24T13:35:03.996 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:03 smithi187 ceph-mon[44911]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi163=b;smithi163=e;smithi163=h;smithi163=k;smithi187=c;smithi187=f;smithi187=i;smithi187=l", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:35:03.996 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:03 smithi187 ceph-mon[44911]: Saving service mds.all spec with placement smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi163=b;smithi163=e;smithi163=h;smithi163=k;smithi187=c;smithi187=f;smithi187=i;smithi187=l;count:12 2024-01-24T13:35:03.996 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:03 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:03.996 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:03 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:35:03.996 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:03 smithi187 ceph-mon[44911]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:03.996 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:03 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:03.996 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:03 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:35:03.996 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:03 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:03.996 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:03 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T13:35:03.996 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:03 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T13:35:03.997 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:03 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:04.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:03 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:35:04.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:03 smithi029 ceph-mon[40651]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:04.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:03 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:04.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:03 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:35:04.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:03 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:04.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:03 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T13:35:04.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:03 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T13:35:04.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:03 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:04.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.146+0000 7f8f9cfd3700 1 -- 172.21.15.29:0/3515702740 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8f9810b440 con 0x7f8f98100e20 2024-01-24T13:35:04.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.146+0000 7f8f877fe700 1 -- 172.21.15.29:0/3515702740 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8f8c099890 con 0x7f8f98100e20 2024-01-24T13:35:04.146 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-01-24T13:35:04.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.149+0000 7f8f9cfd3700 1 -- 172.21.15.29:0/3515702740 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f8f800a0520 msgr2=0x7f8f800a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:04.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.149+0000 7f8f9cfd3700 1 --2- 172.21.15.29:0/3515702740 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f8f800a0520 0x7f8f800a29e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8f88005950 tx=0x7f8f88009260 comp rx=0 tx=0).stop 2024-01-24T13:35:04.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.149+0000 7f8f9cfd3700 1 -- 172.21.15.29:0/3515702740 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8f98100e20 msgr2=0x7f8f98195e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:04.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.149+0000 7f8f9cfd3700 1 --2- 172.21.15.29:0/3515702740 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8f98100e20 0x7f8f98195e60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c009fd0 tx=0x7f8f8c00f690 comp rx=0 tx=0).stop 2024-01-24T13:35:04.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.149+0000 7f8f9cfd3700 1 -- 172.21.15.29:0/3515702740 shutdown_connections 2024-01-24T13:35:04.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.149+0000 7f8f9cfd3700 1 --2- 172.21.15.29:0/3515702740 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f981017c0 0x7f8f9819ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:04.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.149+0000 7f8f9cfd3700 1 --2- 172.21.15.29:0/3515702740 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f8f800a0520 0x7f8f800a29e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:04.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.149+0000 7f8f9cfd3700 1 --2- 172.21.15.29:0/3515702740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f980ffc30 0x7f8f98195920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:04.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.149+0000 7f8f9cfd3700 1 --2- 172.21.15.29:0/3515702740 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8f98100e20 0x7f8f98195e60 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:04.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.149+0000 7f8f9cfd3700 1 -- 172.21.15.29:0/3515702740 >> 172.21.15.29:0/3515702740 conn(0x7f8f980fb1a0 msgr2=0x7f8f980fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:04.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.149+0000 7f8f9cfd3700 1 -- 172.21.15.29:0/3515702740 shutdown_connections 2024-01-24T13:35:04.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.149+0000 7f8f9cfd3700 1 -- 172.21.15.29:0/3515702740 wait complete. 2024-01-24T13:35:04.162 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T13:35:04.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:03 smithi163 ceph-mon[44969]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi163=b;smithi163=e;smithi163=h;smithi163=k;smithi187=c;smithi187=f;smithi187=i;smithi187=l", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T13:35:04.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:03 smithi163 ceph-mon[44969]: Saving service mds.all spec with placement smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi163=b;smithi163=e;smithi163=h;smithi163=k;smithi187=c;smithi187=f;smithi187=i;smithi187=l;count:12 2024-01-24T13:35:04.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:03 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:04.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:03 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:35:04.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:03 smithi163 ceph-mon[44969]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:04.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:03 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:04.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:03 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:35:04.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:03 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:04.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:03 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T13:35:04.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:03 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T13:35:04.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:03 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:04.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.300+0000 7fb26381f700 1 -- 172.21.15.29:0/3686794310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb25c1017e0 msgr2=0x7fb25c108090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:04.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.300+0000 7fb26381f700 1 --2- 172.21.15.29:0/3686794310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb25c1017e0 0x7fb25c108090 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb25400b0d0 tx=0x7fb25400b3e0 comp rx=0 tx=0).stop 2024-01-24T13:35:04.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.300+0000 7fb26381f700 1 -- 172.21.15.29:0/3686794310 shutdown_connections 2024-01-24T13:35:04.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.300+0000 7fb26381f700 1 --2- 172.21.15.29:0/3686794310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb25c1017e0 0x7fb25c108090 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:04.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.300+0000 7fb26381f700 1 --2- 172.21.15.29:0/3686794310 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb25c0ffc50 0x7fb25c100070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:04.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.300+0000 7fb26381f700 1 --2- 172.21.15.29:0/3686794310 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb25c100e40 0x7fb25c1012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:04.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.300+0000 7fb26381f700 1 -- 172.21.15.29:0/3686794310 >> 172.21.15.29:0/3686794310 conn(0x7fb25c0fb1c0 msgr2=0x7fb25c0fd600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:04.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.300+0000 7fb26381f700 1 -- 172.21.15.29:0/3686794310 shutdown_connections 2024-01-24T13:35:04.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.300+0000 7fb26381f700 1 -- 172.21.15.29:0/3686794310 wait complete. 2024-01-24T13:35:04.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.301+0000 7fb26381f700 1 Processor -- start 2024-01-24T13:35:04.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.301+0000 7fb26381f700 1 -- start start 2024-01-24T13:35:04.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.301+0000 7fb26381f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb25c0ffc50 0x7fb25c198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:04.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.301+0000 7fb26381f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb25c100e40 0x7fb25c1991b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:04.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.301+0000 7fb26381f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb25c1017e0 0x7fb25c1a01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:04.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.301+0000 7fb26381f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25c1a07b0 con 0x7fb25c100e40 2024-01-24T13:35:04.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.301+0000 7fb26381f700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25c1a0920 con 0x7fb25c0ffc50 2024-01-24T13:35:04.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.301+0000 7fb26381f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25c1a0ae0 con 0x7fb25c1017e0 2024-01-24T13:35:04.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.301+0000 7fb260dba700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb25c100e40 0x7fb25c1991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:04.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.302+0000 7fb260dba700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb25c100e40 0x7fb25c1991b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55888/0 (socket says 172.21.15.29:55888) 2024-01-24T13:35:04.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.302+0000 7fb260dba700 1 -- 172.21.15.29:0/445708625 learned_addr learned my addr 172.21.15.29:0/445708625 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:04.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.302+0000 7fb261dbc700 1 --2- 172.21.15.29:0/445708625 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb25c1017e0 0x7fb25c1a01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:04.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.302+0000 7fb2615bb700 1 --2- 172.21.15.29:0/445708625 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb25c0ffc50 0x7fb25c198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:04.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.302+0000 7fb260dba700 1 -- 172.21.15.29:0/445708625 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb25c1017e0 msgr2=0x7fb25c1a01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:04.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.302+0000 7fb260dba700 1 --2- 172.21.15.29:0/445708625 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb25c1017e0 0x7fb25c1a01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:04.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.302+0000 7fb260dba700 1 -- 172.21.15.29:0/445708625 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb25c0ffc50 msgr2=0x7fb25c198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:04.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.302+0000 7fb260dba700 1 --2- 172.21.15.29:0/445708625 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb25c0ffc50 0x7fb25c198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:04.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.302+0000 7fb260dba700 1 -- 172.21.15.29:0/445708625 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb254009d00 con 0x7fb25c100e40 2024-01-24T13:35:04.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.302+0000 7fb260dba700 1 --2- 172.21.15.29:0/445708625 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb25c100e40 0x7fb25c1991b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb24c00c9b0 tx=0x7fb24c00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:04.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.302+0000 7fb2527fc700 1 -- 172.21.15.29:0/445708625 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb24c007b00 con 0x7fb25c100e40 2024-01-24T13:35:04.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.302+0000 7fb2527fc700 1 -- 172.21.15.29:0/445708625 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb24c00ce50 con 0x7fb25c100e40 2024-01-24T13:35:04.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.302+0000 7fb26381f700 1 -- 172.21.15.29:0/445708625 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb25c1a0dc0 con 0x7fb25c100e40 2024-01-24T13:35:04.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.302+0000 7fb2527fc700 1 -- 172.21.15.29:0/445708625 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb24c018660 con 0x7fb25c100e40 2024-01-24T13:35:04.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.302+0000 7fb26381f700 1 -- 172.21.15.29:0/445708625 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb25c1a1240 con 0x7fb25c100e40 2024-01-24T13:35:04.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.303+0000 7fb26381f700 1 -- 172.21.15.29:0/445708625 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb25c10b5e0 con 0x7fb25c100e40 2024-01-24T13:35:04.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.304+0000 7fb2527fc700 1 -- 172.21.15.29:0/445708625 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7fb24c00f450 con 0x7fb25c100e40 2024-01-24T13:35:04.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.305+0000 7fb2527fc700 1 --2- 172.21.15.29:0/445708625 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb2480a05f0 0x7fb2480a2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:04.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.305+0000 7fb2615bb700 1 --2- 172.21.15.29:0/445708625 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb2480a05f0 0x7fb2480a2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:04.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.305+0000 7fb2527fc700 1 -- 172.21.15.29:0/445708625 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb24c0cb370 con 0x7fb25c100e40 2024-01-24T13:35:04.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.306+0000 7fb2615bb700 1 --2- 172.21.15.29:0/445708625 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb2480a05f0 0x7fb2480a2ab0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb25c19a140 tx=0x7fb258009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:04.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.309+0000 7fb2527fc700 1 -- 172.21.15.29:0/445708625 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fb24c0953b0 con 0x7fb25c100e40 2024-01-24T13:35:04.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.570+0000 7fb26381f700 1 -- 172.21.15.29:0/445708625 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb25c1a1880 con 0x7fb25c100e40 2024-01-24T13:35:04.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.570+0000 7fb2527fc700 1 -- 172.21.15.29:0/445708625 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+501 (secure 0 0 0) 0x7fb24c007e60 con 0x7fb25c100e40 2024-01-24T13:35:04.571 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:35:04.571 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-01-24T13:35:04.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.573+0000 7fb26381f700 1 -- 172.21.15.29:0/445708625 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb2480a05f0 msgr2=0x7fb2480a2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:04.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.573+0000 7fb26381f700 1 --2- 172.21.15.29:0/445708625 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb2480a05f0 0x7fb2480a2ab0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb25c19a140 tx=0x7fb258009260 comp rx=0 tx=0).stop 2024-01-24T13:35:04.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.573+0000 7fb26381f700 1 -- 172.21.15.29:0/445708625 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb25c100e40 msgr2=0x7fb25c1991b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:04.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.573+0000 7fb26381f700 1 --2- 172.21.15.29:0/445708625 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb25c100e40 0x7fb25c1991b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb24c00c9b0 tx=0x7fb24c00ccc0 comp rx=0 tx=0).stop 2024-01-24T13:35:04.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.573+0000 7fb26381f700 1 -- 172.21.15.29:0/445708625 shutdown_connections 2024-01-24T13:35:04.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.573+0000 7fb26381f700 1 --2- 172.21.15.29:0/445708625 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb25c1017e0 0x7fb25c1a01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:04.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.573+0000 7fb26381f700 1 --2- 172.21.15.29:0/445708625 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb25c0ffc50 0x7fb25c198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:04.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.573+0000 7fb26381f700 1 --2- 172.21.15.29:0/445708625 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb2480a05f0 0x7fb2480a2ab0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:04.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.573+0000 7fb26381f700 1 --2- 172.21.15.29:0/445708625 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb25c100e40 0x7fb25c1991b0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:04.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.573+0000 7fb26381f700 1 -- 172.21.15.29:0/445708625 >> 172.21.15.29:0/445708625 conn(0x7fb25c0fb1c0 msgr2=0x7fb25c068e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:04.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.573+0000 7fb26381f700 1 -- 172.21.15.29:0/445708625 shutdown_connections 2024-01-24T13:35:04.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:04.573+0000 7fb26381f700 1 -- 172.21.15.29:0/445708625 wait complete. 2024-01-24T13:35:04.575 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 1 2024-01-24T13:35:04.643 INFO:journalctl@ceph.mds.l.smithi187.stdout:Jan 24 13:35:04 smithi187 systemd[1]: Starting Ceph mds.l for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:35:04.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:04 smithi029 ceph-mon[40651]: Deploying daemon mds.l on smithi187 2024-01-24T13:35:04.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:04 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/2256076327' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T13:35:04.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:04 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/3515702740' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-24T13:35:04.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:04 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/445708625' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T13:35:05.054 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:04 smithi187 ceph-mon[44911]: Deploying daemon mds.l on smithi187 2024-01-24T13:35:05.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:04 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/2256076327' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T13:35:05.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:04 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/3515702740' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-24T13:35:05.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:04 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/445708625' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T13:35:05.055 INFO:journalctl@ceph.mds.l.smithi187.stdout:Jan 24 13:35:04 smithi187 ceph-mds[80724]: -- 172.21.15.187:0/3977489251 <== mon.1 v2:172.21.15.187:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555ad717f1e0 con 0x555ad7155000 2024-01-24T13:35:05.055 INFO:journalctl@ceph.mds.l.smithi187.stdout:Jan 24 13:35:04 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-l[80701]: starting mds.l at 2024-01-24T13:35:05.055 INFO:journalctl@ceph.mds.l.smithi187.stdout:Jan 24 13:35:04 smithi187 bash[80686]: 2e91733d1897d0b8364d5d39be5faa01d5524520cfc7ad0548bc70da16582233 2024-01-24T13:35:05.055 INFO:journalctl@ceph.mds.l.smithi187.stdout:Jan 24 13:35:04 smithi187 systemd[1]: Started Ceph mds.l for 83afcf5e-babc-11ee-95b1-87774f69a715. 2024-01-24T13:35:05.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:04 smithi163 ceph-mon[44969]: Deploying daemon mds.l on smithi187 2024-01-24T13:35:05.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:04 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/2256076327' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T13:35:05.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:04 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/3515702740' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-24T13:35:05.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:04 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/445708625' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T13:35:05.897 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:05 smithi163 ceph-mon[44969]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:05.897 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:05 smithi163 ceph-mon[44969]: mds.? [v2:172.21.15.187:6832/3688255584,v1:172.21.15.187:6833/3688255584] up:boot 2024-01-24T13:35:05.897 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:05 smithi163 ceph-mon[44969]: fsmap 1 up:standby 2024-01-24T13:35:05.897 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:05 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-01-24T13:35:05.897 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:05 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:05.897 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:05 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:05.897 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:05 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:05.898 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:05 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T13:35:05.898 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:05 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T13:35:05.898 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:05 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:05.898 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:05 smithi163 ceph-mon[44969]: Deploying daemon mds.k on smithi163 2024-01-24T13:35:06.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:05 smithi029 ceph-mon[40651]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:06.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:05 smithi029 ceph-mon[40651]: mds.? [v2:172.21.15.187:6832/3688255584,v1:172.21.15.187:6833/3688255584] up:boot 2024-01-24T13:35:06.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:05 smithi029 ceph-mon[40651]: fsmap 1 up:standby 2024-01-24T13:35:06.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:05 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-01-24T13:35:06.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:05 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:06.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:05 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:06.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:05 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:06.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:05 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T13:35:06.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:05 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T13:35:06.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:05 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:06.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:05 smithi029 ceph-mon[40651]: Deploying daemon mds.k on smithi163 2024-01-24T13:35:06.149 INFO:journalctl@ceph.mds.k.smithi163.stdout:Jan 24 13:35:06 smithi163 systemd[1]: Starting Ceph mds.k for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:35:06.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:05 smithi187 ceph-mon[44911]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:06.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:05 smithi187 ceph-mon[44911]: mds.? [v2:172.21.15.187:6832/3688255584,v1:172.21.15.187:6833/3688255584] up:boot 2024-01-24T13:35:06.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:05 smithi187 ceph-mon[44911]: fsmap 1 up:standby 2024-01-24T13:35:06.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:05 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-01-24T13:35:06.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:05 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:06.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:05 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:06.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:05 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:06.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:05 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T13:35:06.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:05 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T13:35:06.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:05 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:06.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:05 smithi187 ceph-mon[44911]: Deploying daemon mds.k on smithi163 2024-01-24T13:35:06.587 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T13:35:06.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.738+0000 7f6c7965d700 1 -- 172.21.15.29:0/644532731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c74100d60 msgr2=0x7f6c74101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:06.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.738+0000 7f6c7965d700 1 --2- 172.21.15.29:0/644532731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c74100d60 0x7f6c74101180 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c0099c0 tx=0x7f6c5c009cd0 comp rx=0 tx=0).stop 2024-01-24T13:35:06.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.738+0000 7f6c7965d700 1 -- 172.21.15.29:0/644532731 shutdown_connections 2024-01-24T13:35:06.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.738+0000 7f6c7965d700 1 --2- 172.21.15.29:0/644532731 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c741028f0 0x7f6c741091a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:06.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.739+0000 7f6c7965d700 1 --2- 172.21.15.29:0/644532731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c74100d60 0x7f6c74101180 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:06.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.739+0000 7f6c7965d700 1 --2- 172.21.15.29:0/644532731 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c74101f50 0x7f6c741023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:06.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.739+0000 7f6c7965d700 1 -- 172.21.15.29:0/644532731 >> 172.21.15.29:0/644532731 conn(0x7f6c740fc2b0 msgr2=0x7f6c740fe710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:06.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.739+0000 7f6c7965d700 1 -- 172.21.15.29:0/644532731 shutdown_connections 2024-01-24T13:35:06.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.739+0000 7f6c7965d700 1 -- 172.21.15.29:0/644532731 wait complete. 2024-01-24T13:35:06.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.739+0000 7f6c7965d700 1 Processor -- start 2024-01-24T13:35:06.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.739+0000 7f6c7965d700 1 -- start start 2024-01-24T13:35:06.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.740+0000 7f6c7965d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c74100d60 0x7f6c74196a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:06.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.740+0000 7f6c7965d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c74101f50 0x7f6c74196f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:06.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.740+0000 7f6c7965d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c741028f0 0x7f6c7419e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:06.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.740+0000 7f6c7965d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7419e5d0 con 0x7f6c74101f50 2024-01-24T13:35:06.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.740+0000 7f6c7965d700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7419e710 con 0x7f6c741028f0 2024-01-24T13:35:06.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.740+0000 7f6c7965d700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7419e8a0 con 0x7f6c74100d60 2024-01-24T13:35:06.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.740+0000 7f6c727fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c74101f50 0x7f6c74196f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:06.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.740+0000 7f6c727fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c74101f50 0x7f6c74196f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55902/0 (socket says 172.21.15.29:55902) 2024-01-24T13:35:06.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.740+0000 7f6c727fc700 1 -- 172.21.15.29:0/154730026 learned_addr learned my addr 172.21.15.29:0/154730026 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:06.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.740+0000 7f6c737fe700 1 --2- 172.21.15.29:0/154730026 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c741028f0 0x7f6c7419e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:06.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.740+0000 7f6c72ffd700 1 --2- 172.21.15.29:0/154730026 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c74100d60 0x7f6c74196a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:06.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.740+0000 7f6c727fc700 1 -- 172.21.15.29:0/154730026 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c74100d60 msgr2=0x7f6c74196a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:06.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.740+0000 7f6c727fc700 1 --2- 172.21.15.29:0/154730026 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c74100d60 0x7f6c74196a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:06.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.740+0000 7f6c727fc700 1 -- 172.21.15.29:0/154730026 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c741028f0 msgr2=0x7f6c7419e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:06.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.740+0000 7f6c727fc700 1 --2- 172.21.15.29:0/154730026 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c741028f0 0x7f6c7419e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:06.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.740+0000 7f6c727fc700 1 -- 172.21.15.29:0/154730026 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c5c0095f0 con 0x7f6c74101f50 2024-01-24T13:35:06.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.740+0000 7f6c72ffd700 1 --2- 172.21.15.29:0/154730026 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c74100d60 0x7f6c74196a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T13:35:06.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.740+0000 7f6c727fc700 1 --2- 172.21.15.29:0/154730026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c74101f50 0x7f6c74196f90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6c6400e9a0 tx=0x7f6c6400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:06.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.740+0000 7f6c6bfff700 1 -- 172.21.15.29:0/154730026 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c64009790 con 0x7f6c74101f50 2024-01-24T13:35:06.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.741+0000 7f6c6bfff700 1 -- 172.21.15.29:0/154730026 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6c6400cbd0 con 0x7f6c74101f50 2024-01-24T13:35:06.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.741+0000 7f6c7965d700 1 -- 172.21.15.29:0/154730026 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c7419eb80 con 0x7f6c74101f50 2024-01-24T13:35:06.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.741+0000 7f6c6bfff700 1 -- 172.21.15.29:0/154730026 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c64018680 con 0x7f6c74101f50 2024-01-24T13:35:06.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.741+0000 7f6c7965d700 1 -- 172.21.15.29:0/154730026 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c7419f090 con 0x7f6c74101f50 2024-01-24T13:35:06.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.742+0000 7f6c7965d700 1 -- 172.21.15.29:0/154730026 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c741082a0 con 0x7f6c74101f50 2024-01-24T13:35:06.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.743+0000 7f6c6bfff700 1 -- 172.21.15.29:0/154730026 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f6c6400cd40 con 0x7f6c74101f50 2024-01-24T13:35:06.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.743+0000 7f6c6bfff700 1 --2- 172.21.15.29:0/154730026 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f6c600a0570 0x7f6c600a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:06.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.743+0000 7f6c72ffd700 1 --2- 172.21.15.29:0/154730026 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f6c600a0570 0x7f6c600a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:06.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.744+0000 7f6c6bfff700 1 -- 172.21.15.29:0/154730026 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6c64014070 con 0x7f6c74101f50 2024-01-24T13:35:06.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.744+0000 7f6c72ffd700 1 --2- 172.21.15.29:0/154730026 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f6c600a0570 0x7f6c600a2a30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c0098e0 tx=0x7f6c5c019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:06.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:06.748+0000 7f6c6bfff700 1 -- 172.21.15.29:0/154730026 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f6c64052120 con 0x7f6c74101f50 2024-01-24T13:35:06.982 INFO:journalctl@ceph.mds.k.smithi163.stdout:Jan 24 13:35:06 smithi163 ceph-mds[81909]: -- 172.21.15.163:0/2957857711 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560c91a391e0 con 0x560c91a0ec00 2024-01-24T13:35:06.982 INFO:journalctl@ceph.mds.k.smithi163.stdout:Jan 24 13:35:06 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-k[81886]: starting mds.k at 2024-01-24T13:35:06.982 INFO:journalctl@ceph.mds.k.smithi163.stdout:Jan 24 13:35:06 smithi163 bash[81871]: 33630918feef93bccfc8155b64ec6bd8f0a71965ffbf263b49a43b7456b2e279 2024-01-24T13:35:06.982 INFO:journalctl@ceph.mds.k.smithi163.stdout:Jan 24 13:35:06 smithi163 systemd[1]: Started Ceph mds.k for 83afcf5e-babc-11ee-95b1-87774f69a715. 2024-01-24T13:35:07.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:07.013+0000 7f6c7965d700 1 -- 172.21.15.29:0/154730026 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c7402ce20 con 0x7f6c74101f50 2024-01-24T13:35:07.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:07.014+0000 7f6c6bfff700 1 -- 172.21.15.29:0/154730026 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2020 (secure 0 0 0) 0x7f6c640957a0 con 0x7f6c74101f50 2024-01-24T13:35:07.014 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:35:07.014 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3136190517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3136190517},{"type":"v1","addr":"172.21.15.163:6833","nonce":3136190517}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6833/3688255584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":3688255584},{"type":"v1","addr":"172.21.15.187:6833","nonce":3688255584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2024-01-24T13:35:07.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:07.016+0000 7f6c7965d700 1 -- 172.21.15.29:0/154730026 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f6c600a0570 msgr2=0x7f6c600a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:07.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:07.016+0000 7f6c7965d700 1 --2- 172.21.15.29:0/154730026 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f6c600a0570 0x7f6c600a2a30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c0098e0 tx=0x7f6c5c019040 comp rx=0 tx=0).stop 2024-01-24T13:35:07.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:07.016+0000 7f6c7965d700 1 -- 172.21.15.29:0/154730026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c74101f50 msgr2=0x7f6c74196f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:07.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:07.016+0000 7f6c7965d700 1 --2- 172.21.15.29:0/154730026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c74101f50 0x7f6c74196f90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6c6400e9a0 tx=0x7f6c6400ecb0 comp rx=0 tx=0).stop 2024-01-24T13:35:07.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:07.016+0000 7f6c7965d700 1 -- 172.21.15.29:0/154730026 shutdown_connections 2024-01-24T13:35:07.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:07.016+0000 7f6c7965d700 1 --2- 172.21.15.29:0/154730026 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c741028f0 0x7f6c7419e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:07.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:07.016+0000 7f6c7965d700 1 --2- 172.21.15.29:0/154730026 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c74100d60 0x7f6c74196a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:07.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:07.016+0000 7f6c7965d700 1 --2- 172.21.15.29:0/154730026 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f6c600a0570 0x7f6c600a2a30 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:07.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:07.016+0000 7f6c7965d700 1 --2- 172.21.15.29:0/154730026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c74101f50 0x7f6c74196f90 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:07.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:07.016+0000 7f6c7965d700 1 -- 172.21.15.29:0/154730026 >> 172.21.15.29:0/154730026 conn(0x7f6c740fc2b0 msgr2=0x7f6c741053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:07.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:07.016+0000 7f6c7965d700 1 -- 172.21.15.29:0/154730026 shutdown_connections 2024-01-24T13:35:07.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:07.016+0000 7f6c7965d700 1 -- 172.21.15.29:0/154730026 wait complete. 2024-01-24T13:35:07.018 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 3 2024-01-24T13:35:07.651 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:07 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:07.651 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:07 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:07.651 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:07 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:07.652 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:07 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T13:35:07.652 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:07 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T13:35:07.652 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:07 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:07.652 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:07 smithi029 ceph-mon[40651]: Deploying daemon mds.j on smithi029 2024-01-24T13:35:07.652 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:07 smithi029 ceph-mon[40651]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:07.652 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:07 smithi029 ceph-mon[40651]: mds.? [v2:172.21.15.163:6832/3136190517,v1:172.21.15.163:6833/3136190517] up:boot 2024-01-24T13:35:07.652 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:07 smithi029 ceph-mon[40651]: fsmap 2 up:standby 2024-01-24T13:35:07.652 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:07 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-01-24T13:35:07.652 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:07 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/154730026' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T13:35:07.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:07 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:07.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:07 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:07.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:07 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:07.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:07 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T13:35:07.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:07 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T13:35:07.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:07 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:07.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:07 smithi163 ceph-mon[44969]: Deploying daemon mds.j on smithi029 2024-01-24T13:35:07.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:07 smithi163 ceph-mon[44969]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:07.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:07 smithi163 ceph-mon[44969]: mds.? [v2:172.21.15.163:6832/3136190517,v1:172.21.15.163:6833/3136190517] up:boot 2024-01-24T13:35:07.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:07 smithi163 ceph-mon[44969]: fsmap 2 up:standby 2024-01-24T13:35:07.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:07 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-01-24T13:35:07.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:07 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/154730026' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T13:35:08.054 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:07 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:08.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:07 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:08.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:07 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:08.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:07 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T13:35:08.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:07 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T13:35:08.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:07 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:08.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:07 smithi187 ceph-mon[44911]: Deploying daemon mds.j on smithi029 2024-01-24T13:35:08.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:07 smithi187 ceph-mon[44911]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:08.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:07 smithi187 ceph-mon[44911]: mds.? [v2:172.21.15.163:6832/3136190517,v1:172.21.15.163:6833/3136190517] up:boot 2024-01-24T13:35:08.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:07 smithi187 ceph-mon[44911]: fsmap 2 up:standby 2024-01-24T13:35:08.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:07 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-01-24T13:35:08.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:07 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/154730026' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T13:35:08.173 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jan 24 13:35:07 smithi029 systemd[1]: Starting Ceph mds.j for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:35:08.599 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jan 24 13:35:08 smithi029 ceph-mds[91186]: -- 172.21.15.29:0/1951869492 <== mon.0 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556cb43af1e0 con 0x556cb4384800 2024-01-24T13:35:08.599 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jan 24 13:35:08 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-j[91163]: starting mds.j at 2024-01-24T13:35:08.599 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jan 24 13:35:08 smithi029 bash[91148]: 09883734f306e1a2fed5667bb7ae60cc1b0965253049d994394ed0c3144f0e37 2024-01-24T13:35:08.599 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jan 24 13:35:08 smithi029 systemd[1]: Started Ceph mds.j for 83afcf5e-babc-11ee-95b1-87774f69a715. 2024-01-24T13:35:09.031 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T13:35:09.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.187+0000 7f4870661700 1 -- 172.21.15.29:0/1130648568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48681017f0 msgr2=0x7f4868103be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:09.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.187+0000 7f4870661700 1 --2- 172.21.15.29:0/1130648568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48681017f0 0x7f4868103be0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4864009990 tx=0x7f4864009ca0 comp rx=0 tx=0).stop 2024-01-24T13:35:09.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.188+0000 7f4870661700 1 -- 172.21.15.29:0/1130648568 shutdown_connections 2024-01-24T13:35:09.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.188+0000 7f4870661700 1 --2- 172.21.15.29:0/1130648568 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4868106a50 0x7f4868108ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:09.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.188+0000 7f4870661700 1 --2- 172.21.15.29:0/1130648568 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48681017f0 0x7f4868103be0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:09.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.188+0000 7f4870661700 1 --2- 172.21.15.29:0/1130648568 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4868104120 0x7f4868106510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:09.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.188+0000 7f4870661700 1 -- 172.21.15.29:0/1130648568 >> 172.21.15.29:0/1130648568 conn(0x7f48680fb200 msgr2=0x7f48680fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:09.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.188+0000 7f4870661700 1 -- 172.21.15.29:0/1130648568 shutdown_connections 2024-01-24T13:35:09.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.188+0000 7f4870661700 1 -- 172.21.15.29:0/1130648568 wait complete. 2024-01-24T13:35:09.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.188+0000 7f4870661700 1 Processor -- start 2024-01-24T13:35:09.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.188+0000 7f4870661700 1 -- start start 2024-01-24T13:35:09.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.188+0000 7f4870661700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48681017f0 0x7f4868101160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:09.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.188+0000 7f4870661700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4868104120 0x7f48680ff7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:09.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.189+0000 7f4870661700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4868106a50 0x7f48680ffce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:09.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.189+0000 7f4870661700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4868100480 con 0x7f48681017f0 2024-01-24T13:35:09.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.189+0000 7f4870661700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48681005f0 con 0x7f4868106a50 2024-01-24T13:35:09.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.189+0000 7f4870661700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48681a0950 con 0x7f4868104120 2024-01-24T13:35:09.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.189+0000 7f486e3fd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48681017f0 0x7f4868101160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:09.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.189+0000 7f486e3fd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48681017f0 0x7f4868101160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55942/0 (socket says 172.21.15.29:55942) 2024-01-24T13:35:09.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.189+0000 7f486e3fd700 1 -- 172.21.15.29:0/3725811820 learned_addr learned my addr 172.21.15.29:0/3725811820 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:09.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.189+0000 7f486dbfc700 1 --2- 172.21.15.29:0/3725811820 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4868104120 0x7f48680ff7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:09.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.189+0000 7f486ebfe700 1 --2- 172.21.15.29:0/3725811820 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4868106a50 0x7f48680ffce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:09.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.189+0000 7f486e3fd700 1 -- 172.21.15.29:0/3725811820 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4868104120 msgr2=0x7f48680ff7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:09.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.189+0000 7f486e3fd700 1 --2- 172.21.15.29:0/3725811820 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4868104120 0x7f48680ff7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:09.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.189+0000 7f486e3fd700 1 -- 172.21.15.29:0/3725811820 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4868106a50 msgr2=0x7f48680ffce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:09.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.189+0000 7f486e3fd700 1 --2- 172.21.15.29:0/3725811820 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4868106a50 0x7f48680ffce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:09.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.189+0000 7f486e3fd700 1 -- 172.21.15.29:0/3725811820 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48640095f0 con 0x7f48681017f0 2024-01-24T13:35:09.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.189+0000 7f486e3fd700 1 --2- 172.21.15.29:0/3725811820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48681017f0 0x7f4868101160 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f48640098e0 tx=0x7f486400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:09.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.189+0000 7f485f7fe700 1 -- 172.21.15.29:0/3725811820 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486401d070 con 0x7f48681017f0 2024-01-24T13:35:09.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.189+0000 7f485f7fe700 1 -- 172.21.15.29:0/3725811820 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4864022470 con 0x7f48681017f0 2024-01-24T13:35:09.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.189+0000 7f4870661700 1 -- 172.21.15.29:0/3725811820 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48681a0bd0 con 0x7f48681017f0 2024-01-24T13:35:09.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.189+0000 7f485f7fe700 1 -- 172.21.15.29:0/3725811820 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4864017650 con 0x7f48681017f0 2024-01-24T13:35:09.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.189+0000 7f4870661700 1 -- 172.21.15.29:0/3725811820 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48681a1050 con 0x7f48681017f0 2024-01-24T13:35:09.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.191+0000 7f4870661700 1 -- 172.21.15.29:0/3725811820 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48680fcd80 con 0x7f48681017f0 2024-01-24T13:35:09.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.192+0000 7f485f7fe700 1 -- 172.21.15.29:0/3725811820 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f48640225e0 con 0x7f48681017f0 2024-01-24T13:35:09.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.192+0000 7f485f7fe700 1 --2- 172.21.15.29:0/3725811820 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f48540a05f0 0x7f48540a2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:09.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.192+0000 7f486dbfc700 1 --2- 172.21.15.29:0/3725811820 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f48540a05f0 0x7f48540a2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:09.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.192+0000 7f485f7fe700 1 -- 172.21.15.29:0/3725811820 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f48640cc910 con 0x7f48681017f0 2024-01-24T13:35:09.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.193+0000 7f486dbfc700 1 --2- 172.21.15.29:0/3725811820 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f48540a05f0 0x7f48540a2ab0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4858005950 tx=0x7f485800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:09.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.197+0000 7f485f7fe700 1 -- 172.21.15.29:0/3725811820 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f48640969e0 con 0x7f48681017f0 2024-01-24T13:35:09.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.458+0000 7f4870661700 1 -- 172.21.15.29:0/3725811820 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f486802cc70 con 0x7f48681017f0 2024-01-24T13:35:09.458 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:09 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:09.458 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:09 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:09.458 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:09 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:09.458 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:09 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T13:35:09.459 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:09 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T13:35:09.459 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:09 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:09.459 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:09 smithi029 ceph-mon[40651]: Deploying daemon mds.d on smithi029 2024-01-24T13:35:09.459 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:09 smithi029 ceph-mon[40651]: mds.? [v2:172.21.15.29:6834/2401960181,v1:172.21.15.29:6835/2401960181] up:boot 2024-01-24T13:35:09.459 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:09 smithi029 ceph-mon[40651]: fsmap 3 up:standby 2024-01-24T13:35:09.459 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:09 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-01-24T13:35:09.459 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:09 smithi029 ceph-mon[40651]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:09.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.459+0000 7f485f7fe700 1 -- 172.21.15.29:0/3725811820 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2777 (secure 0 0 0) 0x7f486409a000 con 0x7f48681017f0 2024-01-24T13:35:09.459 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:35:09.460 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3136190517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3136190517},{"type":"v1","addr":"172.21.15.163:6833","nonce":3136190517}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6833/3688255584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":3688255584},{"type":"v1","addr":"172.21.15.187:6833","nonce":3688255584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24469,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2401960181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2401960181},{"type":"v1","addr":"172.21.15.29:6835","nonce":2401960181}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4}],"filesystems":[]} 2024-01-24T13:35:09.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.461+0000 7f4870661700 1 -- 172.21.15.29:0/3725811820 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f48540a05f0 msgr2=0x7f48540a2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:09.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.461+0000 7f4870661700 1 --2- 172.21.15.29:0/3725811820 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f48540a05f0 0x7f48540a2ab0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4858005950 tx=0x7f485800b410 comp rx=0 tx=0).stop 2024-01-24T13:35:09.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.461+0000 7f4870661700 1 -- 172.21.15.29:0/3725811820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48681017f0 msgr2=0x7f4868101160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:09.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.461+0000 7f4870661700 1 --2- 172.21.15.29:0/3725811820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48681017f0 0x7f4868101160 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f48640098e0 tx=0x7f486400f690 comp rx=0 tx=0).stop 2024-01-24T13:35:09.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.461+0000 7f4870661700 1 -- 172.21.15.29:0/3725811820 shutdown_connections 2024-01-24T13:35:09.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.461+0000 7f4870661700 1 --2- 172.21.15.29:0/3725811820 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4868106a50 0x7f48680ffce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:09.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.461+0000 7f4870661700 1 --2- 172.21.15.29:0/3725811820 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f48540a05f0 0x7f48540a2ab0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:09.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.461+0000 7f4870661700 1 --2- 172.21.15.29:0/3725811820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48681017f0 0x7f4868101160 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:09.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.461+0000 7f4870661700 1 --2- 172.21.15.29:0/3725811820 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4868104120 0x7f48680ff7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:09.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.461+0000 7f4870661700 1 -- 172.21.15.29:0/3725811820 >> 172.21.15.29:0/3725811820 conn(0x7f48680fb200 msgr2=0x7f4868102460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:09.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.461+0000 7f4870661700 1 -- 172.21.15.29:0/3725811820 shutdown_connections 2024-01-24T13:35:09.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:09.462+0000 7f4870661700 1 -- 172.21.15.29:0/3725811820 wait complete. 2024-01-24T13:35:09.462 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 4 2024-01-24T13:35:09.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:09 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:09.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:09 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:09.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:09 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:09.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:09 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T13:35:09.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:09 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T13:35:09.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:09 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:09.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:09 smithi163 ceph-mon[44969]: Deploying daemon mds.d on smithi029 2024-01-24T13:35:09.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:09 smithi163 ceph-mon[44969]: mds.? [v2:172.21.15.29:6834/2401960181,v1:172.21.15.29:6835/2401960181] up:boot 2024-01-24T13:35:09.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:09 smithi163 ceph-mon[44969]: fsmap 3 up:standby 2024-01-24T13:35:09.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:09 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-01-24T13:35:09.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:09 smithi163 ceph-mon[44969]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:09.804 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:09 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:09.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:09 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:09.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:09 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:09.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:09 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T13:35:09.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:09 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T13:35:09.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:09 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:09.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:09 smithi187 ceph-mon[44911]: Deploying daemon mds.d on smithi029 2024-01-24T13:35:09.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:09 smithi187 ceph-mon[44911]: mds.? [v2:172.21.15.29:6834/2401960181,v1:172.21.15.29:6835/2401960181] up:boot 2024-01-24T13:35:09.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:09 smithi187 ceph-mon[44911]: fsmap 3 up:standby 2024-01-24T13:35:09.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:09 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-01-24T13:35:09.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:09 smithi187 ceph-mon[44911]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:10.008 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jan 24 13:35:09 smithi029 systemd[1]: Starting Ceph mds.d for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:35:10.599 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jan 24 13:35:10 smithi029 bash[91534]: 350e5919ead5451af3b860dbb4a0e38116858ac7abbd7903790d0a9b7cdb97f4 2024-01-24T13:35:10.599 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jan 24 13:35:10 smithi029 ceph-mds[91573]: -- 172.21.15.29:0/3096379090 <== mon.0 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56271198d1e0 con 0x562711962800 2024-01-24T13:35:10.599 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jan 24 13:35:10 smithi029 systemd[1]: Started Ceph mds.d for 83afcf5e-babc-11ee-95b1-87774f69a715. 2024-01-24T13:35:10.599 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jan 24 13:35:10 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-d[91550]: starting mds.d at 2024-01-24T13:35:10.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:10 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/3725811820' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T13:35:10.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:10 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:10.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:10 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:10.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:10 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:10.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:10 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T13:35:10.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:10 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T13:35:10.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:10 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:10.653 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:10 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/3725811820' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T13:35:10.654 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:10 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:10.654 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:10 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:10.654 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:10 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:10.654 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:10 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T13:35:10.654 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:10 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T13:35:10.654 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:10 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:10.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:10 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/3725811820' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T13:35:10.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:10 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:10.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:10 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:10.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:10 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:10.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:10 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T13:35:10.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:10 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T13:35:10.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:10 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:11.485 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T13:35:11.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.637+0000 7fee35685700 1 -- 172.21.15.29:0/2820728641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee300ffc20 msgr2=0x7fee30100040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:11.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.637+0000 7fee35685700 1 --2- 172.21.15.29:0/2820728641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee300ffc20 0x7fee30100040 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fee200099c0 tx=0x7fee20009cd0 comp rx=0 tx=0).stop 2024-01-24T13:35:11.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.637+0000 7fee35685700 1 -- 172.21.15.29:0/2820728641 shutdown_connections 2024-01-24T13:35:11.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.637+0000 7fee35685700 1 --2- 172.21.15.29:0/2820728641 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee301017b0 0x7fee30108060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:11.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.637+0000 7fee35685700 1 --2- 172.21.15.29:0/2820728641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee300ffc20 0x7fee30100040 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:11.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.637+0000 7fee35685700 1 --2- 172.21.15.29:0/2820728641 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fee30100e10 0x7fee30101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:11.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.637+0000 7fee35685700 1 -- 172.21.15.29:0/2820728641 >> 172.21.15.29:0/2820728641 conn(0x7fee300fb170 msgr2=0x7fee300fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:11.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.637+0000 7fee35685700 1 -- 172.21.15.29:0/2820728641 shutdown_connections 2024-01-24T13:35:11.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.637+0000 7fee35685700 1 -- 172.21.15.29:0/2820728641 wait complete. 2024-01-24T13:35:11.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.637+0000 7fee35685700 1 Processor -- start 2024-01-24T13:35:11.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.638+0000 7fee35685700 1 -- start start 2024-01-24T13:35:11.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.638+0000 7fee35685700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee300ffc20 0x7fee30196a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:11.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.638+0000 7fee35685700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fee30100e10 0x7fee30196f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:11.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.638+0000 7fee35685700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee301017b0 0x7fee3019dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:11.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.638+0000 7fee35685700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee3019e5b0 con 0x7fee301017b0 2024-01-24T13:35:11.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.638+0000 7fee35685700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee3019e6f0 con 0x7fee30100e10 2024-01-24T13:35:11.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.638+0000 7fee35685700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee3019e880 con 0x7fee300ffc20 2024-01-24T13:35:11.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.638+0000 7fee2f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee301017b0 0x7fee3019dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:11.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.638+0000 7fee2e7fc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fee30100e10 0x7fee30196f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:11.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.638+0000 7fee2f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee301017b0 0x7fee3019dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56966/0 (socket says 172.21.15.29:56966) 2024-01-24T13:35:11.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.638+0000 7fee2f7fe700 1 -- 172.21.15.29:0/587301086 learned_addr learned my addr 172.21.15.29:0/587301086 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:11.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.638+0000 7fee2effd700 1 --2- 172.21.15.29:0/587301086 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee300ffc20 0x7fee30196a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:11.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.638+0000 7fee2e7fc700 1 -- 172.21.15.29:0/587301086 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee300ffc20 msgr2=0x7fee30196a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:11.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.638+0000 7fee2e7fc700 1 --2- 172.21.15.29:0/587301086 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee300ffc20 0x7fee30196a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:11.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.638+0000 7fee2e7fc700 1 -- 172.21.15.29:0/587301086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee301017b0 msgr2=0x7fee3019dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:11.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.638+0000 7fee2e7fc700 1 --2- 172.21.15.29:0/587301086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee301017b0 0x7fee3019dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:11.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.638+0000 7fee2e7fc700 1 -- 172.21.15.29:0/587301086 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee200095f0 con 0x7fee30100e10 2024-01-24T13:35:11.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.638+0000 7fee2f7fe700 1 --2- 172.21.15.29:0/587301086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee301017b0 0x7fee3019dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T13:35:11.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.639+0000 7fee2e7fc700 1 --2- 172.21.15.29:0/587301086 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fee30100e10 0x7fee30196f70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fee2400e9f0 tx=0x7fee2400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:11.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.639+0000 7fee17fff700 1 -- 172.21.15.29:0/587301086 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee2400c9c0 con 0x7fee30100e10 2024-01-24T13:35:11.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.639+0000 7fee35685700 1 -- 172.21.15.29:0/587301086 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee3019eb60 con 0x7fee30100e10 2024-01-24T13:35:11.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.639+0000 7fee17fff700 1 -- 172.21.15.29:0/587301086 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fee24010430 con 0x7fee30100e10 2024-01-24T13:35:11.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.639+0000 7fee17fff700 1 -- 172.21.15.29:0/587301086 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee24018680 con 0x7fee30100e10 2024-01-24T13:35:11.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.639+0000 7fee35685700 1 -- 172.21.15.29:0/587301086 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee3019f0c0 con 0x7fee30100e10 2024-01-24T13:35:11.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.640+0000 7fee35685700 1 -- 172.21.15.29:0/587301086 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee3010b430 con 0x7fee30100e10 2024-01-24T13:35:11.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.641+0000 7fee17fff700 1 -- 172.21.15.29:0/587301086 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7fee24010860 con 0x7fee30100e10 2024-01-24T13:35:11.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.642+0000 7fee17fff700 1 --2- 172.21.15.29:0/587301086 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fee180a0570 0x7fee180a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:11.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.642+0000 7fee2effd700 1 --2- 172.21.15.29:0/587301086 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fee180a0570 0x7fee180a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:11.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.642+0000 7fee17fff700 1 -- 172.21.15.29:0/587301086 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fee24014070 con 0x7fee30100e10 2024-01-24T13:35:11.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.642+0000 7fee2effd700 1 --2- 172.21.15.29:0/587301086 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fee180a0570 0x7fee180a2a30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fee20009990 tx=0x7fee20005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:11.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.645+0000 7fee17fff700 1 -- 172.21.15.29:0/587301086 <== mon.2 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fee240957a0 con 0x7fee30100e10 2024-01-24T13:35:11.676 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jan 24 13:35:11 smithi163 systemd[1]: Starting Ceph mds.e for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:35:11.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:11 smithi029 ceph-mon[40651]: Deploying daemon mds.e on smithi163 2024-01-24T13:35:11.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:11 smithi029 ceph-mon[40651]: mds.? [v2:172.21.15.29:6836/1295649805,v1:172.21.15.29:6837/1295649805] up:boot 2024-01-24T13:35:11.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:11 smithi029 ceph-mon[40651]: fsmap 4 up:standby 2024-01-24T13:35:11.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:11 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-01-24T13:35:11.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:11 smithi029 ceph-mon[40651]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:11.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.909+0000 7fee35685700 1 -- 172.21.15.29:0/587301086 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee3019f670 con 0x7fee30100e10 2024-01-24T13:35:11.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.910+0000 7fee17fff700 1 -- 172.21.15.29:0/587301086 <== mon.2 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3534 (secure 0 0 0) 0x7fee24098dc0 con 0x7fee30100e10 2024-01-24T13:35:11.910 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:35:11.910 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1295649805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1295649805},{"type":"v1","addr":"172.21.15.29:6837","nonce":1295649805}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3136190517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3136190517},{"type":"v1","addr":"172.21.15.163:6833","nonce":3136190517}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6833/3688255584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":3688255584},{"type":"v1","addr":"172.21.15.187:6833","nonce":3688255584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24469,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2401960181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2401960181},{"type":"v1","addr":"172.21.15.29:6835","nonce":2401960181}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4}],"filesystems":[]} 2024-01-24T13:35:11.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.912+0000 7fee35685700 1 -- 172.21.15.29:0/587301086 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fee180a0570 msgr2=0x7fee180a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:11.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.912+0000 7fee35685700 1 --2- 172.21.15.29:0/587301086 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fee180a0570 0x7fee180a2a30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fee20009990 tx=0x7fee20005fb0 comp rx=0 tx=0).stop 2024-01-24T13:35:11.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.912+0000 7fee35685700 1 -- 172.21.15.29:0/587301086 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fee30100e10 msgr2=0x7fee30196f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:11.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.912+0000 7fee35685700 1 --2- 172.21.15.29:0/587301086 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fee30100e10 0x7fee30196f70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fee2400e9f0 tx=0x7fee2400ed00 comp rx=0 tx=0).stop 2024-01-24T13:35:11.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.912+0000 7fee35685700 1 -- 172.21.15.29:0/587301086 shutdown_connections 2024-01-24T13:35:11.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.912+0000 7fee35685700 1 --2- 172.21.15.29:0/587301086 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fee180a0570 0x7fee180a2a30 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:11.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.912+0000 7fee35685700 1 --2- 172.21.15.29:0/587301086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee301017b0 0x7fee3019dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:11.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.912+0000 7fee35685700 1 --2- 172.21.15.29:0/587301086 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee300ffc20 0x7fee30196a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:11.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.912+0000 7fee35685700 1 --2- 172.21.15.29:0/587301086 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fee30100e10 0x7fee30196f70 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:11.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.912+0000 7fee35685700 1 -- 172.21.15.29:0/587301086 >> 172.21.15.29:0/587301086 conn(0x7fee300fb170 msgr2=0x7fee30067ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:11.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.913+0000 7fee35685700 1 -- 172.21.15.29:0/587301086 shutdown_connections 2024-01-24T13:35:11.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:11.913+0000 7fee35685700 1 -- 172.21.15.29:0/587301086 wait complete. 2024-01-24T13:35:11.913 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 5 2024-01-24T13:35:11.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:11 smithi163 ceph-mon[44969]: Deploying daemon mds.e on smithi163 2024-01-24T13:35:11.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:11 smithi163 ceph-mon[44969]: mds.? [v2:172.21.15.29:6836/1295649805,v1:172.21.15.29:6837/1295649805] up:boot 2024-01-24T13:35:11.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:11 smithi163 ceph-mon[44969]: fsmap 4 up:standby 2024-01-24T13:35:11.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:11 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-01-24T13:35:11.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:11 smithi163 ceph-mon[44969]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:11.982 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jan 24 13:35:11 smithi163 ceph-mds[82260]: -- 172.21.15.163:0/2211417090 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559971dbd1e0 con 0x559971d93000 2024-01-24T13:35:11.982 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jan 24 13:35:11 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-e[82237]: starting mds.e at 2024-01-24T13:35:11.982 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jan 24 13:35:11 smithi163 bash[82222]: 06d87d438787ea46334496a5ba33b4dc77fee42679c54b1bd6710edb274ecba9 2024-01-24T13:35:11.982 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jan 24 13:35:11 smithi163 systemd[1]: Started Ceph mds.e for 83afcf5e-babc-11ee-95b1-87774f69a715. 2024-01-24T13:35:12.054 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:11 smithi187 ceph-mon[44911]: Deploying daemon mds.e on smithi163 2024-01-24T13:35:12.054 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:11 smithi187 ceph-mon[44911]: mds.? [v2:172.21.15.29:6836/1295649805,v1:172.21.15.29:6837/1295649805] up:boot 2024-01-24T13:35:12.054 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:11 smithi187 ceph-mon[44911]: fsmap 4 up:standby 2024-01-24T13:35:12.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:11 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-01-24T13:35:12.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:11 smithi187 ceph-mon[44911]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:12.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:12 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/587301086' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T13:35:12.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:12 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:12.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:12 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:12.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:12 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:12.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:12 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T13:35:12.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:12 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T13:35:12.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:12 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:12.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:12 smithi163 ceph-mon[44969]: Deploying daemon mds.f on smithi187 2024-01-24T13:35:13.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:12 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/587301086' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T13:35:13.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:12 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:13.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:12 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:13.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:12 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:13.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:12 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T13:35:13.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:12 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T13:35:13.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:12 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:13.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:12 smithi187 ceph-mon[44911]: Deploying daemon mds.f on smithi187 2024-01-24T13:35:13.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:12 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/587301086' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T13:35:13.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:12 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:13.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:12 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:13.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:12 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:13.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:12 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T13:35:13.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:12 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T13:35:13.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:12 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:13.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:12 smithi029 ceph-mon[40651]: Deploying daemon mds.f on smithi187 2024-01-24T13:35:13.257 INFO:journalctl@ceph.mds.f.smithi187.stdout:Jan 24 13:35:13 smithi187 systemd[1]: Starting Ceph mds.f for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:35:13.927 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T13:35:13.959 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:13 smithi029 ceph-mon[40651]: mds.? [v2:172.21.15.163:6834/2665026288,v1:172.21.15.163:6835/2665026288] up:boot 2024-01-24T13:35:13.959 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:13 smithi029 ceph-mon[40651]: fsmap 5 up:standby 2024-01-24T13:35:13.959 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:13 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-01-24T13:35:13.959 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:13 smithi029 ceph-mon[40651]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:13.959 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:13 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:13.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:13 smithi163 ceph-mon[44969]: mds.? [v2:172.21.15.163:6834/2665026288,v1:172.21.15.163:6835/2665026288] up:boot 2024-01-24T13:35:13.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:13 smithi163 ceph-mon[44969]: fsmap 5 up:standby 2024-01-24T13:35:13.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:13 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-01-24T13:35:13.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:13 smithi163 ceph-mon[44969]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:13.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:13 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:14.054 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:13 smithi187 ceph-mon[44911]: mds.? [v2:172.21.15.163:6834/2665026288,v1:172.21.15.163:6835/2665026288] up:boot 2024-01-24T13:35:14.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:13 smithi187 ceph-mon[44911]: fsmap 5 up:standby 2024-01-24T13:35:14.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:13 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-01-24T13:35:14.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:13 smithi187 ceph-mon[44911]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:14.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:13 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:14.055 INFO:journalctl@ceph.mds.f.smithi187.stdout:Jan 24 13:35:13 smithi187 bash[81027]: 9727c41794770c174bda83e051c8b77e43686a4b7708caa0cc2b83ad007eaf46 2024-01-24T13:35:14.055 INFO:journalctl@ceph.mds.f.smithi187.stdout:Jan 24 13:35:13 smithi187 systemd[1]: Started Ceph mds.f for 83afcf5e-babc-11ee-95b1-87774f69a715. 2024-01-24T13:35:14.055 INFO:journalctl@ceph.mds.f.smithi187.stdout:Jan 24 13:35:13 smithi187 ceph-mds[81065]: -- 172.21.15.187:0/3650962845 <== mon.1 v2:172.21.15.187:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c1629f11e0 con 0x55c1629c7000 2024-01-24T13:35:14.055 INFO:journalctl@ceph.mds.f.smithi187.stdout:Jan 24 13:35:13 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-f[81042]: starting mds.f at 2024-01-24T13:35:14.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.073+0000 7f0602aa1700 1 -- 172.21.15.29:0/3867259639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05fc106a00 msgr2=0x7f05fc108e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:14.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.073+0000 7f0602aa1700 1 --2- 172.21.15.29:0/3867259639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05fc106a00 0x7f05fc108e70 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f05f400b0a0 tx=0x7f05f400b3b0 comp rx=0 tx=0).stop 2024-01-24T13:35:14.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.073+0000 7f0602aa1700 1 -- 172.21.15.29:0/3867259639 shutdown_connections 2024-01-24T13:35:14.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.073+0000 7f0602aa1700 1 --2- 172.21.15.29:0/3867259639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05fc106a00 0x7f05fc108e70 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:14.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.073+0000 7f0602aa1700 1 --2- 172.21.15.29:0/3867259639 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05fc1017a0 0x7f05fc103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:14.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.073+0000 7f0602aa1700 1 --2- 172.21.15.29:0/3867259639 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f05fc1040d0 0x7f05fc1064c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:14.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.073+0000 7f0602aa1700 1 -- 172.21.15.29:0/3867259639 >> 172.21.15.29:0/3867259639 conn(0x7f05fc0fb170 msgr2=0x7f05fc0fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:14.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.074+0000 7f0602aa1700 1 -- 172.21.15.29:0/3867259639 shutdown_connections 2024-01-24T13:35:14.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.074+0000 7f0602aa1700 1 -- 172.21.15.29:0/3867259639 wait complete. 2024-01-24T13:35:14.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.074+0000 7f0602aa1700 1 Processor -- start 2024-01-24T13:35:14.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.074+0000 7f0602aa1700 1 -- start start 2024-01-24T13:35:14.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.074+0000 7f0602aa1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f05fc1017a0 0x7f05fc196af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:14.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.074+0000 7f0602aa1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05fc1040d0 0x7f05fc197030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:14.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.074+0000 7f0602aa1700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05fc106a00 0x7f05fc19e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:14.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.074+0000 7f0602aa1700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc19e670 con 0x7f05fc1040d0 2024-01-24T13:35:14.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.074+0000 7f0602aa1700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc19e7b0 con 0x7f05fc106a00 2024-01-24T13:35:14.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.074+0000 7f0602aa1700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc19e940 con 0x7f05fc1017a0 2024-01-24T13:35:14.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.075+0000 7f05fbfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05fc1040d0 0x7f05fc197030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:14.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.075+0000 7f05fbfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05fc1040d0 0x7f05fc197030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56998/0 (socket says 172.21.15.29:56998) 2024-01-24T13:35:14.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.075+0000 7f05fbfff700 1 -- 172.21.15.29:0/438866328 learned_addr learned my addr 172.21.15.29:0/438866328 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:14.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.075+0000 7f060083d700 1 --2- 172.21.15.29:0/438866328 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f05fc1017a0 0x7f05fc196af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:14.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.075+0000 7f060103e700 1 --2- 172.21.15.29:0/438866328 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05fc106a00 0x7f05fc19e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:14.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.075+0000 7f05fbfff700 1 -- 172.21.15.29:0/438866328 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f05fc1017a0 msgr2=0x7f05fc196af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:14.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.075+0000 7f05fbfff700 1 --2- 172.21.15.29:0/438866328 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f05fc1017a0 0x7f05fc196af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:14.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.075+0000 7f05fbfff700 1 -- 172.21.15.29:0/438866328 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05fc106a00 msgr2=0x7f05fc19e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:14.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.075+0000 7f05fbfff700 1 --2- 172.21.15.29:0/438866328 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05fc106a00 0x7f05fc19e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:14.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.075+0000 7f05fbfff700 1 -- 172.21.15.29:0/438866328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f4009d00 con 0x7f05fc1040d0 2024-01-24T13:35:14.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.075+0000 7f05fbfff700 1 --2- 172.21.15.29:0/438866328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05fc1040d0 0x7f05fc197030 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f05f000d950 tx=0x7f05f000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:14.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.075+0000 7f05f9ffb700 1 -- 172.21.15.29:0/438866328 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f000f890 con 0x7f05fc1040d0 2024-01-24T13:35:14.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.075+0000 7f05f9ffb700 1 -- 172.21.15.29:0/438866328 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f05f0010460 con 0x7f05fc1040d0 2024-01-24T13:35:14.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.075+0000 7f0602aa1700 1 -- 172.21.15.29:0/438866328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05fc19ec20 con 0x7f05fc1040d0 2024-01-24T13:35:14.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.075+0000 7f05f9ffb700 1 -- 172.21.15.29:0/438866328 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f000e610 con 0x7f05fc1040d0 2024-01-24T13:35:14.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.076+0000 7f0602aa1700 1 -- 172.21.15.29:0/438866328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05fc19f180 con 0x7f05fc1040d0 2024-01-24T13:35:14.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.077+0000 7f0602aa1700 1 -- 172.21.15.29:0/438866328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05fc0fccf0 con 0x7f05fc1040d0 2024-01-24T13:35:14.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.078+0000 7f05f9ffb700 1 -- 172.21.15.29:0/438866328 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f05f00108d0 con 0x7f05fc1040d0 2024-01-24T13:35:14.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.078+0000 7f05f9ffb700 1 --2- 172.21.15.29:0/438866328 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f05e40a05f0 0x7f05e40a2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:14.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.078+0000 7f060083d700 1 --2- 172.21.15.29:0/438866328 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f05e40a05f0 0x7f05e40a2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:14.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.078+0000 7f05f9ffb700 1 -- 172.21.15.29:0/438866328 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f05f00cb200 con 0x7f05fc1040d0 2024-01-24T13:35:14.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.079+0000 7f060083d700 1 --2- 172.21.15.29:0/438866328 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f05e40a05f0 0x7f05e40a2ab0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f05ec005950 tx=0x7f05ec00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:14.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.083+0000 7f05f9ffb700 1 -- 172.21.15.29:0/438866328 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f05f0095240 con 0x7f05fc1040d0 2024-01-24T13:35:14.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.346+0000 7f0602aa1700 1 -- 172.21.15.29:0/438866328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05fc02cc70 con 0x7f05fc1040d0 2024-01-24T13:35:14.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.346+0000 7f05f9ffb700 1 -- 172.21.15.29:0/438866328 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5054 (secure 0 0 0) 0x7f05f0098860 con 0x7f05fc1040d0 2024-01-24T13:35:14.347 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:35:14.347 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1295649805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1295649805},{"type":"v1","addr":"172.21.15.29:6837","nonce":1295649805}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14559,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2665026288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2665026288},{"type":"v1","addr":"172.21.15.163:6835","nonce":2665026288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3136190517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3136190517},{"type":"v1","addr":"172.21.15.163:6833","nonce":3136190517}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6833/3688255584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":3688255584},{"type":"v1","addr":"172.21.15.187:6833","nonce":3688255584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24469,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2401960181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2401960181},{"type":"v1","addr":"172.21.15.29:6835","nonce":2401960181}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/1709179808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1709179808},{"type":"v1","addr":"172.21.15.187:6835","nonce":1709179808}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[]} 2024-01-24T13:35:14.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.349+0000 7f0602aa1700 1 -- 172.21.15.29:0/438866328 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f05e40a05f0 msgr2=0x7f05e40a2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:14.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.349+0000 7f0602aa1700 1 --2- 172.21.15.29:0/438866328 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f05e40a05f0 0x7f05e40a2ab0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f05ec005950 tx=0x7f05ec00b410 comp rx=0 tx=0).stop 2024-01-24T13:35:14.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.349+0000 7f0602aa1700 1 -- 172.21.15.29:0/438866328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05fc1040d0 msgr2=0x7f05fc197030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:14.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.349+0000 7f0602aa1700 1 --2- 172.21.15.29:0/438866328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05fc1040d0 0x7f05fc197030 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f05f000d950 tx=0x7f05f000dc60 comp rx=0 tx=0).stop 2024-01-24T13:35:14.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.349+0000 7f0602aa1700 1 -- 172.21.15.29:0/438866328 shutdown_connections 2024-01-24T13:35:14.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.349+0000 7f0602aa1700 1 --2- 172.21.15.29:0/438866328 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05fc106a00 0x7f05fc19e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:14.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.349+0000 7f0602aa1700 1 --2- 172.21.15.29:0/438866328 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f05fc1017a0 0x7f05fc196af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:14.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.349+0000 7f0602aa1700 1 --2- 172.21.15.29:0/438866328 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f05e40a05f0 0x7f05e40a2ab0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:14.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.349+0000 7f0602aa1700 1 --2- 172.21.15.29:0/438866328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05fc1040d0 0x7f05fc197030 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:14.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.349+0000 7f0602aa1700 1 -- 172.21.15.29:0/438866328 >> 172.21.15.29:0/438866328 conn(0x7f05fc0fb170 msgr2=0x7f05fc102410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:14.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.349+0000 7f0602aa1700 1 -- 172.21.15.29:0/438866328 shutdown_connections 2024-01-24T13:35:14.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:14.349+0000 7f0602aa1700 1 -- 172.21.15.29:0/438866328 wait complete. 2024-01-24T13:35:14.350 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 7 2024-01-24T13:35:14.949 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:14 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:14.950 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:14 smithi187 ceph-mon[44911]: mds.? [v2:172.21.15.187:6834/1709179808,v1:172.21.15.187:6835/1709179808] up:boot 2024-01-24T13:35:14.950 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:14 smithi187 ceph-mon[44911]: fsmap 6 up:standby 2024-01-24T13:35:14.950 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:14 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-01-24T13:35:14.950 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:14 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:14.950 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:14 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T13:35:14.950 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:14 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T13:35:14.950 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:14 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:14.950 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:14 smithi187 ceph-mon[44911]: Deploying daemon mds.c on smithi187 2024-01-24T13:35:14.950 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:14 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/438866328' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T13:35:14.950 INFO:journalctl@ceph.mds.c.smithi187.stdout:Jan 24 13:35:14 smithi187 systemd[1]: Starting Ceph mds.c for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:35:14.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:14 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:14.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:14 smithi163 ceph-mon[44969]: mds.? [v2:172.21.15.187:6834/1709179808,v1:172.21.15.187:6835/1709179808] up:boot 2024-01-24T13:35:14.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:14 smithi163 ceph-mon[44969]: fsmap 6 up:standby 2024-01-24T13:35:14.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:14 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-01-24T13:35:14.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:14 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:14.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:14 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T13:35:14.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:14 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T13:35:14.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:14 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:14.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:14 smithi163 ceph-mon[44969]: Deploying daemon mds.c on smithi187 2024-01-24T13:35:14.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:14 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/438866328' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T13:35:15.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:14 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:15.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:14 smithi029 ceph-mon[40651]: mds.? [v2:172.21.15.187:6834/1709179808,v1:172.21.15.187:6835/1709179808] up:boot 2024-01-24T13:35:15.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:14 smithi029 ceph-mon[40651]: fsmap 6 up:standby 2024-01-24T13:35:15.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:14 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-01-24T13:35:15.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:14 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:15.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:14 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T13:35:15.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:14 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T13:35:15.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:14 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:15.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:14 smithi029 ceph-mon[40651]: Deploying daemon mds.c on smithi187 2024-01-24T13:35:15.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:14 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/438866328' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T13:35:15.695 INFO:journalctl@ceph.mds.c.smithi187.stdout:Jan 24 13:35:15 smithi187 ceph-mds[81404]: -- 172.21.15.187:0/2416148325 <== mon.1 v2:172.21.15.187:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556774c171e0 con 0x556774becc00 2024-01-24T13:35:15.695 INFO:journalctl@ceph.mds.c.smithi187.stdout:Jan 24 13:35:15 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-c[81381]: starting mds.c at 2024-01-24T13:35:15.695 INFO:journalctl@ceph.mds.c.smithi187.stdout:Jan 24 13:35:15 smithi187 bash[81367]: 21c73b134ad5f5474aba291c677295139af4980084f32662dd422a587ffdf7b9 2024-01-24T13:35:15.695 INFO:journalctl@ceph.mds.c.smithi187.stdout:Jan 24 13:35:15 smithi187 systemd[1]: Started Ceph mds.c for 83afcf5e-babc-11ee-95b1-87774f69a715. 2024-01-24T13:35:15.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:15 smithi163 ceph-mon[44969]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:15.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:15 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:15.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:15 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:15.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:15 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:15.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:15 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T13:35:15.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:15 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T13:35:15.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:15 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:16.054 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:15 smithi187 ceph-mon[44911]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:16.054 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:15 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:16.054 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:15 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:16.054 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:15 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:16.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:15 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T13:35:16.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:15 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T13:35:16.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:15 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:16.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:15 smithi029 ceph-mon[40651]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:16.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:15 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:16.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:15 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:16.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:15 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:16.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:15 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T13:35:16.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:15 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T13:35:16.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:15 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:16.363 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T13:35:16.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.511+0000 7efec5d24700 1 -- 172.21.15.29:0/3844141439 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efec01040d0 msgr2=0x7efec01064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:16.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.511+0000 7efec5d24700 1 --2- 172.21.15.29:0/3844141439 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efec01040d0 0x7efec01064c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7efeb4009990 tx=0x7efeb4009ca0 comp rx=0 tx=0).stop 2024-01-24T13:35:16.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.511+0000 7efec5d24700 1 -- 172.21.15.29:0/3844141439 shutdown_connections 2024-01-24T13:35:16.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.511+0000 7efec5d24700 1 --2- 172.21.15.29:0/3844141439 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efec0106a00 0x7efec0108e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:16.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.511+0000 7efec5d24700 1 --2- 172.21.15.29:0/3844141439 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efec01017a0 0x7efec0103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:16.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.511+0000 7efec5d24700 1 --2- 172.21.15.29:0/3844141439 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efec01040d0 0x7efec01064c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:16.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.511+0000 7efec5d24700 1 -- 172.21.15.29:0/3844141439 >> 172.21.15.29:0/3844141439 conn(0x7efec00fb170 msgr2=0x7efec00fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:16.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.511+0000 7efec5d24700 1 -- 172.21.15.29:0/3844141439 shutdown_connections 2024-01-24T13:35:16.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.511+0000 7efec5d24700 1 -- 172.21.15.29:0/3844141439 wait complete. 2024-01-24T13:35:16.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.512+0000 7efec5d24700 1 Processor -- start 2024-01-24T13:35:16.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.512+0000 7efec5d24700 1 -- start start 2024-01-24T13:35:16.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.512+0000 7efec5d24700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efec01017a0 0x7efec0198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:16.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.512+0000 7efec5d24700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efec01040d0 0x7efec0199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:16.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.512+0000 7efec5d24700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efec0106a00 0x7efec01a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:16.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.512+0000 7efec5d24700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec01a0770 con 0x7efec01040d0 2024-01-24T13:35:16.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.512+0000 7efec5d24700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec01a08e0 con 0x7efec0106a00 2024-01-24T13:35:16.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.512+0000 7efec5d24700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec01a0aa0 con 0x7efec01017a0 2024-01-24T13:35:16.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.513+0000 7efebeffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efec01040d0 0x7efec0199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:16.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.513+0000 7efebffff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efec0106a00 0x7efec01a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:16.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.513+0000 7efebeffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efec01040d0 0x7efec0199170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57016/0 (socket says 172.21.15.29:57016) 2024-01-24T13:35:16.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.513+0000 7efebeffd700 1 -- 172.21.15.29:0/573840112 learned_addr learned my addr 172.21.15.29:0/573840112 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:16.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.513+0000 7efebf7fe700 1 --2- 172.21.15.29:0/573840112 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efec01017a0 0x7efec0198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:16.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.513+0000 7efebeffd700 1 -- 172.21.15.29:0/573840112 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efec01017a0 msgr2=0x7efec0198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:16.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.513+0000 7efebeffd700 1 --2- 172.21.15.29:0/573840112 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efec01017a0 0x7efec0198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:16.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.513+0000 7efebeffd700 1 -- 172.21.15.29:0/573840112 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efec0106a00 msgr2=0x7efec01a01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:16.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.513+0000 7efebeffd700 1 --2- 172.21.15.29:0/573840112 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efec0106a00 0x7efec01a01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:16.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.513+0000 7efebeffd700 1 -- 172.21.15.29:0/573840112 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb40095f0 con 0x7efec01040d0 2024-01-24T13:35:16.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.513+0000 7efebffff700 1 --2- 172.21.15.29:0/573840112 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efec0106a00 0x7efec01a01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T13:35:16.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.513+0000 7efebeffd700 1 --2- 172.21.15.29:0/573840112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efec01040d0 0x7efec0199170 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7efeb4004730 tx=0x7efeb4004760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:16.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.513+0000 7efebcff9700 1 -- 172.21.15.29:0/573840112 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb401d070 con 0x7efec01040d0 2024-01-24T13:35:16.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.513+0000 7efec5d24700 1 -- 172.21.15.29:0/573840112 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efec01a0d20 con 0x7efec01040d0 2024-01-24T13:35:16.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.513+0000 7efebcff9700 1 -- 172.21.15.29:0/573840112 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efeb4004c90 con 0x7efec01040d0 2024-01-24T13:35:16.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.513+0000 7efebcff9700 1 -- 172.21.15.29:0/573840112 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb40054b0 con 0x7efec01040d0 2024-01-24T13:35:16.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.513+0000 7efec5d24700 1 -- 172.21.15.29:0/573840112 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efec01a10e0 con 0x7efec01040d0 2024-01-24T13:35:16.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.514+0000 7efec5d24700 1 -- 172.21.15.29:0/573840112 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efec01085d0 con 0x7efec01040d0 2024-01-24T13:35:16.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.516+0000 7efebcff9700 1 -- 172.21.15.29:0/573840112 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7efeb400fbc0 con 0x7efec01040d0 2024-01-24T13:35:16.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.516+0000 7efebcff9700 1 --2- 172.21.15.29:0/573840112 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7efea80a0570 0x7efea80a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:16.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.516+0000 7efebf7fe700 1 --2- 172.21.15.29:0/573840112 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7efea80a0570 0x7efea80a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:16.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.516+0000 7efebcff9700 1 -- 172.21.15.29:0/573840112 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7efeb40cc780 con 0x7efec01040d0 2024-01-24T13:35:16.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.517+0000 7efebf7fe700 1 --2- 172.21.15.29:0/573840112 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7efea80a0570 0x7efea80a2a30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7efec019a100 tx=0x7efeb0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:16.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.521+0000 7efebcff9700 1 -- 172.21.15.29:0/573840112 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7efeb405b120 con 0x7efec01040d0 2024-01-24T13:35:16.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.781+0000 7efec5d24700 1 -- 172.21.15.29:0/573840112 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efec002cc70 con 0x7efec01040d0 2024-01-24T13:35:16.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.782+0000 7efebcff9700 1 -- 172.21.15.29:0/573840112 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+5811 (secure 0 0 0) 0x7efeb4096850 con 0x7efec01040d0 2024-01-24T13:35:16.782 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:35:16.782 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1295649805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1295649805},{"type":"v1","addr":"172.21.15.29:6837","nonce":1295649805}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14559,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2665026288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2665026288},{"type":"v1","addr":"172.21.15.163:6835","nonce":2665026288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3136190517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3136190517},{"type":"v1","addr":"172.21.15.163:6833","nonce":3136190517}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6833/3688255584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":3688255584},{"type":"v1","addr":"172.21.15.187:6833","nonce":3688255584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24469,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2401960181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2401960181},{"type":"v1","addr":"172.21.15.29:6835","nonce":2401960181}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/1709179808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1709179808},{"type":"v1","addr":"172.21.15.187:6835","nonce":1709179808}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/533724911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":533724911},{"type":"v1","addr":"172.21.15.187:6837","nonce":533724911}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2024-01-24T13:35:16.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.784+0000 7efec5d24700 1 -- 172.21.15.29:0/573840112 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7efea80a0570 msgr2=0x7efea80a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:16.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.784+0000 7efec5d24700 1 --2- 172.21.15.29:0/573840112 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7efea80a0570 0x7efea80a2a30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7efec019a100 tx=0x7efeb0009260 comp rx=0 tx=0).stop 2024-01-24T13:35:16.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.784+0000 7efec5d24700 1 -- 172.21.15.29:0/573840112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efec01040d0 msgr2=0x7efec0199170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:16.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.784+0000 7efec5d24700 1 --2- 172.21.15.29:0/573840112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efec01040d0 0x7efec0199170 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7efeb4004730 tx=0x7efeb4004760 comp rx=0 tx=0).stop 2024-01-24T13:35:16.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.784+0000 7efec5d24700 1 -- 172.21.15.29:0/573840112 shutdown_connections 2024-01-24T13:35:16.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.784+0000 7efec5d24700 1 --2- 172.21.15.29:0/573840112 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efec0106a00 0x7efec01a01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:16.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.784+0000 7efec5d24700 1 --2- 172.21.15.29:0/573840112 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efec01017a0 0x7efec0198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:16.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.784+0000 7efec5d24700 1 --2- 172.21.15.29:0/573840112 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7efea80a0570 0x7efea80a2a30 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:16.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.784+0000 7efec5d24700 1 --2- 172.21.15.29:0/573840112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efec01040d0 0x7efec0199170 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:16.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.784+0000 7efec5d24700 1 -- 172.21.15.29:0/573840112 >> 172.21.15.29:0/573840112 conn(0x7efec00fb170 msgr2=0x7efec00ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:16.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.785+0000 7efec5d24700 1 -- 172.21.15.29:0/573840112 shutdown_connections 2024-01-24T13:35:16.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:16.785+0000 7efec5d24700 1 -- 172.21.15.29:0/573840112 wait complete. 2024-01-24T13:35:16.785 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 8 2024-01-24T13:35:16.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:16 smithi163 ceph-mon[44969]: Deploying daemon mds.h on smithi163 2024-01-24T13:35:16.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:16 smithi163 ceph-mon[44969]: mds.? [v2:172.21.15.187:6836/533724911,v1:172.21.15.187:6837/533724911] up:boot 2024-01-24T13:35:16.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:16 smithi163 ceph-mon[44969]: fsmap 7 up:standby 2024-01-24T13:35:16.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:16 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-01-24T13:35:16.983 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jan 24 13:35:16 smithi163 systemd[1]: Starting Ceph mds.h for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:35:17.054 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:16 smithi187 ceph-mon[44911]: Deploying daemon mds.h on smithi163 2024-01-24T13:35:17.054 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:16 smithi187 ceph-mon[44911]: mds.? [v2:172.21.15.187:6836/533724911,v1:172.21.15.187:6837/533724911] up:boot 2024-01-24T13:35:17.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:16 smithi187 ceph-mon[44911]: fsmap 7 up:standby 2024-01-24T13:35:17.055 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:16 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-01-24T13:35:17.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:16 smithi029 ceph-mon[40651]: Deploying daemon mds.h on smithi163 2024-01-24T13:35:17.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:16 smithi029 ceph-mon[40651]: mds.? [v2:172.21.15.187:6836/533724911,v1:172.21.15.187:6837/533724911] up:boot 2024-01-24T13:35:17.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:16 smithi029 ceph-mon[40651]: fsmap 7 up:standby 2024-01-24T13:35:17.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:16 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-01-24T13:35:17.482 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jan 24 13:35:17 smithi163 ceph-mds[82598]: -- 172.21.15.163:0/2072602940 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562e4cd1d1e0 con 0x562e4ccf2800 2024-01-24T13:35:17.482 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jan 24 13:35:17 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-h[82574]: starting mds.h at 2024-01-24T13:35:17.482 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jan 24 13:35:17 smithi163 bash[82560]: cbda95481aff55f7ea98ab7cb7a573afc5a49708a81d9bfb0f255d63bd604c98 2024-01-24T13:35:17.482 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jan 24 13:35:17 smithi163 systemd[1]: Started Ceph mds.h for 83afcf5e-babc-11ee-95b1-87774f69a715. 2024-01-24T13:35:17.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:17 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/573840112' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T13:35:17.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:17 smithi163 ceph-mon[44969]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:17.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:17 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:17.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:17 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:17.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:17 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:17.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:17 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T13:35:17.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:17 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T13:35:17.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:17 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:18.048 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:17 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/573840112' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T13:35:18.048 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:17 smithi187 ceph-mon[44911]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:18.048 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:17 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:18.048 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:17 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:18.048 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:17 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:18.048 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:17 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T13:35:18.049 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:17 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T13:35:18.049 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:17 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:18.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:17 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/573840112' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T13:35:18.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:17 smithi029 ceph-mon[40651]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:18.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:17 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:18.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:17 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:18.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:17 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:18.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:17 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T13:35:18.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:17 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T13:35:18.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:17 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:18.798 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T13:35:18.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:18.945+0000 7f92e5583700 1 -- 172.21.15.29:0/2081810073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92e0100d40 msgr2=0x7f92e0101160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:18.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:18.945+0000 7f92e5583700 1 --2- 172.21.15.29:0/2081810073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92e0100d40 0x7f92e0101160 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f92d00099c0 tx=0x7f92d0009cd0 comp rx=0 tx=0).stop 2024-01-24T13:35:18.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:18.945+0000 7f92e5583700 1 -- 172.21.15.29:0/2081810073 shutdown_connections 2024-01-24T13:35:18.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:18.945+0000 7f92e5583700 1 --2- 172.21.15.29:0/2081810073 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92e01028d0 0x7f92e0109180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:18.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:18.945+0000 7f92e5583700 1 --2- 172.21.15.29:0/2081810073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92e0100d40 0x7f92e0101160 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:18.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:18.945+0000 7f92e5583700 1 --2- 172.21.15.29:0/2081810073 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92e0101f30 0x7f92e0102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:18.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:18.945+0000 7f92e5583700 1 -- 172.21.15.29:0/2081810073 >> 172.21.15.29:0/2081810073 conn(0x7f92e00fc290 msgr2=0x7f92e00fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:18.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:18.945+0000 7f92e5583700 1 -- 172.21.15.29:0/2081810073 shutdown_connections 2024-01-24T13:35:18.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:18.945+0000 7f92e5583700 1 -- 172.21.15.29:0/2081810073 wait complete. 2024-01-24T13:35:18.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:18.946+0000 7f92e5583700 1 Processor -- start 2024-01-24T13:35:18.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:18.946+0000 7f92e5583700 1 -- start start 2024-01-24T13:35:18.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:18.946+0000 7f92e5583700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92e0100d40 0x7f92e0198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:18.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:18.946+0000 7f92e5583700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92e0101f30 0x7f92e01991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:18.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:18.946+0000 7f92e5583700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92e01028d0 0x7f92e01a01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:18.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:18.946+0000 7f92e5583700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e01a07c0 con 0x7f92e0101f30 2024-01-24T13:35:18.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:18.946+0000 7f92e5583700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e01a0930 con 0x7f92e0100d40 2024-01-24T13:35:18.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:18.946+0000 7f92e5583700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e01a0af0 con 0x7f92e01028d0 2024-01-24T13:35:18.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:18.946+0000 7f92d67fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92e0101f30 0x7f92e01991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:18.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:18.946+0000 7f92d67fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92e0101f30 0x7f92e01991c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57036/0 (socket says 172.21.15.29:57036) 2024-01-24T13:35:18.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:18.946+0000 7f92d67fc700 1 -- 172.21.15.29:0/3631855873 learned_addr learned my addr 172.21.15.29:0/3631855873 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:18.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:18.946+0000 7f92deffd700 1 --2- 172.21.15.29:0/3631855873 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92e0100d40 0x7f92e0198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:18.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:18.946+0000 7f92df7fe700 1 --2- 172.21.15.29:0/3631855873 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92e01028d0 0x7f92e01a01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:18.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:18.947+0000 7f92d67fc700 1 -- 172.21.15.29:0/3631855873 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92e01028d0 msgr2=0x7f92e01a01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:18.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:18.947+0000 7f92d67fc700 1 --2- 172.21.15.29:0/3631855873 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92e01028d0 0x7f92e01a01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:18.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:18.947+0000 7f92d67fc700 1 -- 172.21.15.29:0/3631855873 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92e0100d40 msgr2=0x7f92e0198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:18.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:18.947+0000 7f92d67fc700 1 --2- 172.21.15.29:0/3631855873 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92e0100d40 0x7f92e0198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:18.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:18.947+0000 7f92d67fc700 1 -- 172.21.15.29:0/3631855873 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d00095f0 con 0x7f92e0101f30 2024-01-24T13:35:18.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:18.947+0000 7f92d67fc700 1 --2- 172.21.15.29:0/3631855873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92e0101f30 0x7f92e01991c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f92c800d950 tx=0x7f92c800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:18.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:18.947+0000 7f92dcff9700 1 -- 172.21.15.29:0/3631855873 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c800f890 con 0x7f92e0101f30 2024-01-24T13:35:18.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:18.947+0000 7f92dcff9700 1 -- 172.21.15.29:0/3631855873 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f92c8010460 con 0x7f92e0101f30 2024-01-24T13:35:18.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:18.947+0000 7f92e5583700 1 -- 172.21.15.29:0/3631855873 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92e01a0dd0 con 0x7f92e0101f30 2024-01-24T13:35:18.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:18.947+0000 7f92dcff9700 1 -- 172.21.15.29:0/3631855873 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c800e610 con 0x7f92e0101f30 2024-01-24T13:35:18.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:18.947+0000 7f92e5583700 1 -- 172.21.15.29:0/3631855873 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92e01a1250 con 0x7f92e0101f30 2024-01-24T13:35:18.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:18.948+0000 7f92e5583700 1 -- 172.21.15.29:0/3631855873 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92e01933c0 con 0x7f92e0101f30 2024-01-24T13:35:18.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:18.949+0000 7f92dcff9700 1 -- 172.21.15.29:0/3631855873 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f92c80108d0 con 0x7f92e0101f30 2024-01-24T13:35:18.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:18.950+0000 7f92dcff9700 1 --2- 172.21.15.29:0/3631855873 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f92cc0a05f0 0x7f92cc0a2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:18.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:18.950+0000 7f92deffd700 1 --2- 172.21.15.29:0/3631855873 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f92cc0a05f0 0x7f92cc0a2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:18.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:18.950+0000 7f92dcff9700 1 -- 172.21.15.29:0/3631855873 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f92c80cb2c0 con 0x7f92e0101f30 2024-01-24T13:35:18.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:18.950+0000 7f92deffd700 1 --2- 172.21.15.29:0/3631855873 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f92cc0a05f0 0x7f92cc0a2ab0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f92e019a150 tx=0x7f92d000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:18.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:18.954+0000 7f92dcff9700 1 -- 172.21.15.29:0/3631855873 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f92c8095260 con 0x7f92e0101f30 2024-01-24T13:35:18.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:18 smithi163 ceph-mon[44969]: Deploying daemon mds.i on smithi187 2024-01-24T13:35:18.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:18 smithi163 ceph-mon[44969]: mds.? [v2:172.21.15.163:6836/1610317406,v1:172.21.15.163:6837/1610317406] up:boot 2024-01-24T13:35:18.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:18 smithi163 ceph-mon[44969]: fsmap 8 up:standby 2024-01-24T13:35:18.982 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:18 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-01-24T13:35:19.017 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:18 smithi187 ceph-mon[44911]: Deploying daemon mds.i on smithi187 2024-01-24T13:35:19.017 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:18 smithi187 ceph-mon[44911]: mds.? [v2:172.21.15.163:6836/1610317406,v1:172.21.15.163:6837/1610317406] up:boot 2024-01-24T13:35:19.017 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:18 smithi187 ceph-mon[44911]: fsmap 8 up:standby 2024-01-24T13:35:19.017 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:18 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-01-24T13:35:19.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:18 smithi029 ceph-mon[40651]: Deploying daemon mds.i on smithi187 2024-01-24T13:35:19.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:18 smithi029 ceph-mon[40651]: mds.? [v2:172.21.15.163:6836/1610317406,v1:172.21.15.163:6837/1610317406] up:boot 2024-01-24T13:35:19.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:18 smithi029 ceph-mon[40651]: fsmap 8 up:standby 2024-01-24T13:35:19.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:18 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-01-24T13:35:19.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:19.225+0000 7f92e5583700 1 -- 172.21.15.29:0/3631855873 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92e002cc70 con 0x7f92e0101f30 2024-01-24T13:35:19.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:19.226+0000 7f92dcff9700 1 -- 172.21.15.29:0/3631855873 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6571 (secure 0 0 0) 0x7f92c8098880 con 0x7f92e0101f30 2024-01-24T13:35:19.226 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:35:19.226 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1295649805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1295649805},{"type":"v1","addr":"172.21.15.29:6837","nonce":1295649805}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14559,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2665026288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2665026288},{"type":"v1","addr":"172.21.15.163:6835","nonce":2665026288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3136190517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3136190517},{"type":"v1","addr":"172.21.15.163:6833","nonce":3136190517}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6833/3688255584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":3688255584},{"type":"v1","addr":"172.21.15.187:6833","nonce":3688255584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24469,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2401960181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2401960181},{"type":"v1","addr":"172.21.15.29:6835","nonce":2401960181}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/1709179808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1709179808},{"type":"v1","addr":"172.21.15.187:6835","nonce":1709179808}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/533724911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":533724911},{"type":"v1","addr":"172.21.15.187:6837","nonce":533724911}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/1610317406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":1610317406},{"type":"v1","addr":"172.21.15.163:6837","nonce":1610317406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2024-01-24T13:35:19.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:19.228+0000 7f92e5583700 1 -- 172.21.15.29:0/3631855873 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f92cc0a05f0 msgr2=0x7f92cc0a2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:19.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:19.228+0000 7f92e5583700 1 --2- 172.21.15.29:0/3631855873 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f92cc0a05f0 0x7f92cc0a2ab0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f92e019a150 tx=0x7f92d000b540 comp rx=0 tx=0).stop 2024-01-24T13:35:19.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:19.228+0000 7f92e5583700 1 -- 172.21.15.29:0/3631855873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92e0101f30 msgr2=0x7f92e01991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:19.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:19.228+0000 7f92e5583700 1 --2- 172.21.15.29:0/3631855873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92e0101f30 0x7f92e01991c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f92c800d950 tx=0x7f92c800dc60 comp rx=0 tx=0).stop 2024-01-24T13:35:19.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:19.228+0000 7f92e5583700 1 -- 172.21.15.29:0/3631855873 shutdown_connections 2024-01-24T13:35:19.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:19.228+0000 7f92e5583700 1 --2- 172.21.15.29:0/3631855873 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92e01028d0 0x7f92e01a01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:19.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:19.228+0000 7f92e5583700 1 --2- 172.21.15.29:0/3631855873 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92e0100d40 0x7f92e0198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:19.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:19.228+0000 7f92e5583700 1 --2- 172.21.15.29:0/3631855873 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f92cc0a05f0 0x7f92cc0a2ab0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:19.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:19.228+0000 7f92e5583700 1 --2- 172.21.15.29:0/3631855873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92e0101f30 0x7f92e01991c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:19.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:19.228+0000 7f92e5583700 1 -- 172.21.15.29:0/3631855873 >> 172.21.15.29:0/3631855873 conn(0x7f92e00fc290 msgr2=0x7f92e0107820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:19.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:19.229+0000 7f92e5583700 1 -- 172.21.15.29:0/3631855873 shutdown_connections 2024-01-24T13:35:19.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:19.229+0000 7f92e5583700 1 -- 172.21.15.29:0/3631855873 wait complete. 2024-01-24T13:35:19.230 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 9 2024-01-24T13:35:19.304 INFO:journalctl@ceph.mds.i.smithi187.stdout:Jan 24 13:35:19 smithi187 bash[81708]: ace4861033f778ea400c307a419e837b98e4844e6992f6cf75cf2783924b5e9b 2024-01-24T13:35:19.305 INFO:journalctl@ceph.mds.i.smithi187.stdout:Jan 24 13:35:19 smithi187 systemd[1]: Started Ceph mds.i for 83afcf5e-babc-11ee-95b1-87774f69a715. 2024-01-24T13:35:20.204 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:20 smithi029 ceph-mon[40651]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:20.204 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:20 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:20.204 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:20 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:20.204 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:20 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:20.204 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:20 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T13:35:20.204 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:20 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T13:35:20.204 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:20 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:20.204 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:20 smithi029 ceph-mon[40651]: Deploying daemon mds.g on smithi029 2024-01-24T13:35:20.204 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:20 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/3631855873' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T13:35:20.204 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:20 smithi029 ceph-mon[40651]: mds.? [v2:172.21.15.187:6838/3209015343,v1:172.21.15.187:6839/3209015343] up:boot 2024-01-24T13:35:20.205 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:20 smithi029 ceph-mon[40651]: fsmap 9 up:standby 2024-01-24T13:35:20.205 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:20 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-01-24T13:35:20.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:20 smithi163 ceph-mon[44969]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:20.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:20 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:20.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:20 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:20.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:20 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:20.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:20 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T13:35:20.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:20 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T13:35:20.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:20 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:20.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:20 smithi163 ceph-mon[44969]: Deploying daemon mds.g on smithi029 2024-01-24T13:35:20.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:20 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/3631855873' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T13:35:20.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:20 smithi163 ceph-mon[44969]: mds.? [v2:172.21.15.187:6838/3209015343,v1:172.21.15.187:6839/3209015343] up:boot 2024-01-24T13:35:20.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:20 smithi163 ceph-mon[44969]: fsmap 9 up:standby 2024-01-24T13:35:20.483 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:20 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-01-24T13:35:20.554 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:20 smithi187 ceph-mon[44911]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:20.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:20 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:20.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:20 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:20.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:20 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:20.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:20 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T13:35:20.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:20 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T13:35:20.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:20 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:20.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:20 smithi187 ceph-mon[44911]: Deploying daemon mds.g on smithi029 2024-01-24T13:35:20.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:20 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/3631855873' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T13:35:20.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:20 smithi187 ceph-mon[44911]: mds.? [v2:172.21.15.187:6838/3209015343,v1:172.21.15.187:6839/3209015343] up:boot 2024-01-24T13:35:20.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:20 smithi187 ceph-mon[44911]: fsmap 9 up:standby 2024-01-24T13:35:20.555 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:20 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-01-24T13:35:20.849 INFO:journalctl@ceph.mds.g.smithi029.stdout:Jan 24 13:35:20 smithi029 systemd[1]: Starting Ceph mds.g for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:35:21.244 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T13:35:21.276 INFO:journalctl@ceph.mds.g.smithi029.stdout:Jan 24 13:35:20 smithi029 bash[92055]: b6ddae2ab1d2597a64d6c5f3d39e0a7c891b4bb926e1de3711aa462ca7aa37d4 2024-01-24T13:35:21.277 INFO:journalctl@ceph.mds.g.smithi029.stdout:Jan 24 13:35:20 smithi029 ceph-mds[92093]: -- 172.21.15.29:0/2718994650 <== mon.0 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55594be3f1e0 con 0x55594be15000 2024-01-24T13:35:21.277 INFO:journalctl@ceph.mds.g.smithi029.stdout:Jan 24 13:35:20 smithi029 systemd[1]: Started Ceph mds.g for 83afcf5e-babc-11ee-95b1-87774f69a715. 2024-01-24T13:35:21.277 INFO:journalctl@ceph.mds.g.smithi029.stdout:Jan 24 13:35:20 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-g[92070]: starting mds.g at 2024-01-24T13:35:21.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.397+0000 7f84853bd700 1 -- 172.21.15.29:0/1133544354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8480104160 msgr2=0x7f8480106550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:21.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.397+0000 7f84853bd700 1 --2- 172.21.15.29:0/1133544354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8480104160 0x7f8480106550 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8468009990 tx=0x7f8468009ca0 comp rx=0 tx=0).stop 2024-01-24T13:35:21.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.397+0000 7f84853bd700 1 -- 172.21.15.29:0/1133544354 shutdown_connections 2024-01-24T13:35:21.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.397+0000 7f84853bd700 1 --2- 172.21.15.29:0/1133544354 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8480106a90 0x7f8480108ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:21.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.397+0000 7f84853bd700 1 --2- 172.21.15.29:0/1133544354 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8480101830 0x7f8480103c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:21.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.397+0000 7f84853bd700 1 --2- 172.21.15.29:0/1133544354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8480104160 0x7f8480106550 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:21.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.397+0000 7f84853bd700 1 -- 172.21.15.29:0/1133544354 >> 172.21.15.29:0/1133544354 conn(0x7f84800fb1a0 msgr2=0x7f84800fd600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:21.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.397+0000 7f84853bd700 1 -- 172.21.15.29:0/1133544354 shutdown_connections 2024-01-24T13:35:21.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.397+0000 7f84853bd700 1 -- 172.21.15.29:0/1133544354 wait complete. 2024-01-24T13:35:21.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.398+0000 7f84853bd700 1 Processor -- start 2024-01-24T13:35:21.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.398+0000 7f84853bd700 1 -- start start 2024-01-24T13:35:21.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.398+0000 7f84853bd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8480101830 0x7f84801969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:21.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.398+0000 7f84853bd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8480104160 0x7f8480196ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:21.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.398+0000 7f84853bd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8480106a90 0x7f848019df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:21.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.398+0000 7f84853bd700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f848019e520 con 0x7f8480101830 2024-01-24T13:35:21.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.398+0000 7f84853bd700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f848019e660 con 0x7f8480104160 2024-01-24T13:35:21.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.398+0000 7f84853bd700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f848019e7f0 con 0x7f8480106a90 2024-01-24T13:35:21.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.399+0000 7f847effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8480101830 0x7f84801969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:21.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.399+0000 7f847effd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8480101830 0x7f84801969a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59110/0 (socket says 172.21.15.29:59110) 2024-01-24T13:35:21.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.399+0000 7f847effd700 1 -- 172.21.15.29:0/2776479192 learned_addr learned my addr 172.21.15.29:0/2776479192 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:21.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.399+0000 7f847e7fc700 1 --2- 172.21.15.29:0/2776479192 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8480104160 0x7f8480196ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:21.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.399+0000 7f847f7fe700 1 --2- 172.21.15.29:0/2776479192 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8480106a90 0x7f848019df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:21.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.399+0000 7f847effd700 1 -- 172.21.15.29:0/2776479192 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8480106a90 msgr2=0x7f848019df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:21.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.399+0000 7f847effd700 1 --2- 172.21.15.29:0/2776479192 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8480106a90 0x7f848019df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:21.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.399+0000 7f847effd700 1 -- 172.21.15.29:0/2776479192 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8480104160 msgr2=0x7f8480196ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:21.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.399+0000 7f847effd700 1 --2- 172.21.15.29:0/2776479192 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8480104160 0x7f8480196ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:21.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.399+0000 7f847effd700 1 -- 172.21.15.29:0/2776479192 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84680095f0 con 0x7f8480101830 2024-01-24T13:35:21.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.399+0000 7f847effd700 1 --2- 172.21.15.29:0/2776479192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8480101830 0x7f84801969a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f847000e9a0 tx=0x7f847000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:21.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.399+0000 7f8477fff700 1 -- 172.21.15.29:0/2776479192 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8470009790 con 0x7f8480101830 2024-01-24T13:35:21.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.399+0000 7f8477fff700 1 -- 172.21.15.29:0/2776479192 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8470010430 con 0x7f8480101830 2024-01-24T13:35:21.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.399+0000 7f84853bd700 1 -- 172.21.15.29:0/2776479192 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f848019ead0 con 0x7f8480101830 2024-01-24T13:35:21.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.399+0000 7f8477fff700 1 -- 172.21.15.29:0/2776479192 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8470018680 con 0x7f8480101830 2024-01-24T13:35:21.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.399+0000 7f84853bd700 1 -- 172.21.15.29:0/2776479192 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f848019efe0 con 0x7f8480101830 2024-01-24T13:35:21.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.401+0000 7f84853bd700 1 -- 172.21.15.29:0/2776479192 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84800fcd20 con 0x7f8480101830 2024-01-24T13:35:21.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.402+0000 7f8477fff700 1 -- 172.21.15.29:0/2776479192 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f84700105a0 con 0x7f8480101830 2024-01-24T13:35:21.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.403+0000 7f8477fff700 1 --2- 172.21.15.29:0/2776479192 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f846c0a0570 0x7f846c0a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:21.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.403+0000 7f847e7fc700 1 --2- 172.21.15.29:0/2776479192 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f846c0a0570 0x7f846c0a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:21.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.403+0000 7f8477fff700 1 -- 172.21.15.29:0/2776479192 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f8470014070 con 0x7f8480101830 2024-01-24T13:35:21.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.404+0000 7f847e7fc700 1 --2- 172.21.15.29:0/2776479192 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f846c0a0570 0x7f846c0a2a30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f846800b2e0 tx=0x7f8468005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:21.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.407+0000 7f8477fff700 1 -- 172.21.15.29:0/2776479192 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f84700958b0 con 0x7f8480101830 2024-01-24T13:35:21.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.707+0000 7f84853bd700 1 -- 172.21.15.29:0/2776479192 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f848002cc70 con 0x7f8480101830 2024-01-24T13:35:21.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.708+0000 7f8477fff700 1 -- 172.21.15.29:0/2776479192 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7333 (secure 0 0 0) 0x7f8470098ed0 con 0x7f8480101830 2024-01-24T13:35:21.708 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:35:21.708 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1295649805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1295649805},{"type":"v1","addr":"172.21.15.29:6837","nonce":1295649805}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14559,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2665026288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2665026288},{"type":"v1","addr":"172.21.15.163:6835","nonce":2665026288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3136190517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3136190517},{"type":"v1","addr":"172.21.15.163:6833","nonce":3136190517}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6833/3688255584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":3688255584},{"type":"v1","addr":"172.21.15.187:6833","nonce":3688255584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24469,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2401960181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2401960181},{"type":"v1","addr":"172.21.15.29:6835","nonce":2401960181}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/1709179808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1709179808},{"type":"v1","addr":"172.21.15.187:6835","nonce":1709179808}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/533724911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":533724911},{"type":"v1","addr":"172.21.15.187:6837","nonce":533724911}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/1610317406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":1610317406},{"type":"v1","addr":"172.21.15.163:6837","nonce":1610317406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24523,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6839/3209015343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6838","nonce":3209015343},{"type":"v1","addr":"172.21.15.187:6839","nonce":3209015343}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2024-01-24T13:35:21.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.710+0000 7f84853bd700 1 -- 172.21.15.29:0/2776479192 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f846c0a0570 msgr2=0x7f846c0a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:21.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.710+0000 7f84853bd700 1 --2- 172.21.15.29:0/2776479192 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f846c0a0570 0x7f846c0a2a30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f846800b2e0 tx=0x7f8468005fb0 comp rx=0 tx=0).stop 2024-01-24T13:35:21.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.710+0000 7f84853bd700 1 -- 172.21.15.29:0/2776479192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8480101830 msgr2=0x7f84801969a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:21.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.710+0000 7f84853bd700 1 --2- 172.21.15.29:0/2776479192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8480101830 0x7f84801969a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f847000e9a0 tx=0x7f847000ecb0 comp rx=0 tx=0).stop 2024-01-24T13:35:21.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.710+0000 7f84853bd700 1 -- 172.21.15.29:0/2776479192 shutdown_connections 2024-01-24T13:35:21.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.710+0000 7f84853bd700 1 --2- 172.21.15.29:0/2776479192 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8480106a90 0x7f848019df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:21.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.710+0000 7f84853bd700 1 --2- 172.21.15.29:0/2776479192 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f846c0a0570 0x7f846c0a2a30 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:21.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.710+0000 7f84853bd700 1 --2- 172.21.15.29:0/2776479192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8480101830 0x7f84801969a0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:21.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.710+0000 7f84853bd700 1 --2- 172.21.15.29:0/2776479192 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8480104160 0x7f8480196ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:21.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.710+0000 7f84853bd700 1 -- 172.21.15.29:0/2776479192 >> 172.21.15.29:0/2776479192 conn(0x7f84800fb1a0 msgr2=0x7f84801024a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:21.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.710+0000 7f84853bd700 1 -- 172.21.15.29:0/2776479192 shutdown_connections 2024-01-24T13:35:21.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:21.711+0000 7f84853bd700 1 -- 172.21.15.29:0/2776479192 wait complete. 2024-01-24T13:35:21.711 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 10 2024-01-24T13:35:21.931 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:21 smithi029 ceph-mon[40651]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:21.931 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:21 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:21.931 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:21 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:21.931 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:21 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:21.931 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:21 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T13:35:22.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:21 smithi163 ceph-mon[44969]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:22.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:21 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:22.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:21 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:22.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:21 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:22.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:21 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T13:35:22.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:21 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T13:35:22.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:21 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:22.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:21 smithi163 ceph-mon[44969]: Deploying daemon mds.a on smithi029 2024-01-24T13:35:22.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:21 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/2776479192' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T13:35:22.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:21 smithi163 ceph-mon[44969]: mds.? [v2:172.21.15.29:6838/581075460,v1:172.21.15.29:6839/581075460] up:boot 2024-01-24T13:35:22.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:21 smithi163 ceph-mon[44969]: fsmap 10 up:standby 2024-01-24T13:35:22.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:21 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-01-24T13:35:22.243 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:21 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T13:35:22.243 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:21 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:22.243 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:21 smithi029 ceph-mon[40651]: Deploying daemon mds.a on smithi029 2024-01-24T13:35:22.243 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:21 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/2776479192' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T13:35:22.243 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:21 smithi029 ceph-mon[40651]: mds.? [v2:172.21.15.29:6838/581075460,v1:172.21.15.29:6839/581075460] up:boot 2024-01-24T13:35:22.243 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:21 smithi029 ceph-mon[40651]: fsmap 10 up:standby 2024-01-24T13:35:22.243 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:21 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-01-24T13:35:22.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:21 smithi187 ceph-mon[44911]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:22.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:21 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:22.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:21 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:22.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:21 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:22.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:21 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T13:35:22.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:21 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T13:35:22.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:21 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:22.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:21 smithi187 ceph-mon[44911]: Deploying daemon mds.a on smithi029 2024-01-24T13:35:22.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:21 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/2776479192' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T13:35:22.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:21 smithi187 ceph-mon[44911]: mds.? [v2:172.21.15.29:6838/581075460,v1:172.21.15.29:6839/581075460] up:boot 2024-01-24T13:35:22.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:21 smithi187 ceph-mon[44911]: fsmap 10 up:standby 2024-01-24T13:35:22.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:21 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-01-24T13:35:22.600 INFO:journalctl@ceph.mds.a.smithi029.stdout:Jan 24 13:35:22 smithi029 systemd[1]: Starting Ceph mds.a for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:35:23.099 INFO:journalctl@ceph.mds.a.smithi029.stdout:Jan 24 13:35:22 smithi029 ceph-mds[92482]: -- 172.21.15.29:0/4142793099 <== mon.0 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c0faf511e0 con 0x55c0faf26800 2024-01-24T13:35:23.099 INFO:journalctl@ceph.mds.a.smithi029.stdout:Jan 24 13:35:22 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-a[92459]: starting mds.a at 2024-01-24T13:35:23.099 INFO:journalctl@ceph.mds.a.smithi029.stdout:Jan 24 13:35:22 smithi029 bash[92444]: 0298d6785e540b9d188458bd4dd3c1e08a5b9f11b8f40cc69c92f8ae13ffe004 2024-01-24T13:35:23.099 INFO:journalctl@ceph.mds.a.smithi029.stdout:Jan 24 13:35:22 smithi029 systemd[1]: Started Ceph mds.a for 83afcf5e-babc-11ee-95b1-87774f69a715. 2024-01-24T13:35:23.724 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T13:35:23.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:23.879+0000 7fd6ad352700 1 -- 172.21.15.29:0/50853548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6a8102900 msgr2=0x7fd6a81091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:23.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:23.879+0000 7fd6ad352700 1 --2- 172.21.15.29:0/50853548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6a8102900 0x7fd6a81091b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd6a000b0d0 tx=0x7fd6a000b3e0 comp rx=0 tx=0).stop 2024-01-24T13:35:23.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:23.879+0000 7fd6ad352700 1 -- 172.21.15.29:0/50853548 shutdown_connections 2024-01-24T13:35:23.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:23.879+0000 7fd6ad352700 1 --2- 172.21.15.29:0/50853548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6a8102900 0x7fd6a81091b0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:23.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:23.879+0000 7fd6ad352700 1 --2- 172.21.15.29:0/50853548 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6a8100d70 0x7fd6a8101190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:23.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:23.879+0000 7fd6ad352700 1 --2- 172.21.15.29:0/50853548 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd6a8101f60 0x7fd6a81023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:23.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:23.879+0000 7fd6ad352700 1 -- 172.21.15.29:0/50853548 >> 172.21.15.29:0/50853548 conn(0x7fd6a80fc2c0 msgr2=0x7fd6a80fe720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:23.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:23.879+0000 7fd6ad352700 1 -- 172.21.15.29:0/50853548 shutdown_connections 2024-01-24T13:35:23.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:23.879+0000 7fd6ad352700 1 -- 172.21.15.29:0/50853548 wait complete. 2024-01-24T13:35:23.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:23.880+0000 7fd6ad352700 1 Processor -- start 2024-01-24T13:35:23.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:23.880+0000 7fd6ad352700 1 -- start start 2024-01-24T13:35:23.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:23.880+0000 7fd6ad352700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6a8100d70 0x7fd6a8074af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:23.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:23.880+0000 7fd6ad352700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd6a8101f60 0x7fd6a8073130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:23.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:23.881+0000 7fd6ad352700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6a8102900 0x7fd6a8073670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:23.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:23.881+0000 7fd6ad352700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a8073e10 con 0x7fd6a8100d70 2024-01-24T13:35:23.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:23.881+0000 7fd6ad352700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a8073f80 con 0x7fd6a8102900 2024-01-24T13:35:23.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:23.881+0000 7fd6ad352700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a81a0b20 con 0x7fd6a8101f60 2024-01-24T13:35:23.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:23.881+0000 7fd6a6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6a8100d70 0x7fd6a8074af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:23.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:23.881+0000 7fd6a6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6a8100d70 0x7fd6a8074af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59140/0 (socket says 172.21.15.29:59140) 2024-01-24T13:35:23.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:23.881+0000 7fd6a6ffd700 1 -- 172.21.15.29:0/21667989 learned_addr learned my addr 172.21.15.29:0/21667989 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:23.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:23.881+0000 7fd6a67fc700 1 --2- 172.21.15.29:0/21667989 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd6a8101f60 0x7fd6a8073130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:23.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:23.881+0000 7fd6a77fe700 1 --2- 172.21.15.29:0/21667989 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6a8102900 0x7fd6a8073670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:23.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:23.881+0000 7fd6a6ffd700 1 -- 172.21.15.29:0/21667989 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd6a8101f60 msgr2=0x7fd6a8073130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:23.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:23.881+0000 7fd6a6ffd700 1 --2- 172.21.15.29:0/21667989 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd6a8101f60 0x7fd6a8073130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:23.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:23.881+0000 7fd6a6ffd700 1 -- 172.21.15.29:0/21667989 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6a8102900 msgr2=0x7fd6a8073670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:23.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:23.881+0000 7fd6a6ffd700 1 --2- 172.21.15.29:0/21667989 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6a8102900 0x7fd6a8073670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:23.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:23.881+0000 7fd6a6ffd700 1 -- 172.21.15.29:0/21667989 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a0009d00 con 0x7fd6a8100d70 2024-01-24T13:35:23.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:23.881+0000 7fd6a6ffd700 1 --2- 172.21.15.29:0/21667989 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6a8100d70 0x7fd6a8074af0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd69000e970 tx=0x7fd69000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:23.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:23.881+0000 7fd69ffff700 1 -- 172.21.15.29:0/21667989 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd690009790 con 0x7fd6a8100d70 2024-01-24T13:35:23.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:23.881+0000 7fd6ad352700 1 -- 172.21.15.29:0/21667989 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6a81a0e00 con 0x7fd6a8100d70 2024-01-24T13:35:23.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:23.881+0000 7fd69ffff700 1 -- 172.21.15.29:0/21667989 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd69000eec0 con 0x7fd6a8100d70 2024-01-24T13:35:23.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:23.881+0000 7fd69ffff700 1 -- 172.21.15.29:0/21667989 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd690003cf0 con 0x7fd6a8100d70 2024-01-24T13:35:23.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:23.881+0000 7fd6ad352700 1 -- 172.21.15.29:0/21667989 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6a81a1280 con 0x7fd6a8100d70 2024-01-24T13:35:23.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:23.882+0000 7fd6ad352700 1 -- 172.21.15.29:0/21667989 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6a810c5e0 con 0x7fd6a8100d70 2024-01-24T13:35:23.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:23.884+0000 7fd69ffff700 1 -- 172.21.15.29:0/21667989 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7fd69000ccd0 con 0x7fd6a8100d70 2024-01-24T13:35:23.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:23.884+0000 7fd69ffff700 1 --2- 172.21.15.29:0/21667989 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fd6940a0570 0x7fd6940a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:23.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:23.885+0000 7fd6a67fc700 1 --2- 172.21.15.29:0/21667989 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fd6940a0570 0x7fd6940a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:23.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:23.885+0000 7fd69ffff700 1 -- 172.21.15.29:0/21667989 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd690014070 con 0x7fd6a8100d70 2024-01-24T13:35:23.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:23.885+0000 7fd6a67fc700 1 --2- 172.21.15.29:0/21667989 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fd6940a0570 0x7fd6940a2a30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd698005950 tx=0x7fd69800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:23.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:23.888+0000 7fd69ffff700 1 -- 172.21.15.29:0/21667989 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fd690095930 con 0x7fd6a8100d70 2024-01-24T13:35:24.066 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:23 smithi163 ceph-mon[44969]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:24.066 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:23 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:24.066 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:23 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:24.066 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:23 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:24.066 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:23 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T13:35:24.066 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:23 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T13:35:24.066 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:23 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:24.066 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:23 smithi163 ceph-mon[44969]: Deploying daemon mds.b on smithi163 2024-01-24T13:35:24.067 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:23 smithi163 ceph-mon[44969]: mds.? [v2:172.21.15.29:6840/146067403,v1:172.21.15.29:6841/146067403] up:boot 2024-01-24T13:35:24.067 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:23 smithi163 ceph-mon[44969]: fsmap 11 up:standby 2024-01-24T13:35:24.067 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:23 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-01-24T13:35:24.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:23 smithi029 ceph-mon[40651]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:24.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:23 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:24.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:23 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:24.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:23 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:24.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:23 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T13:35:24.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:23 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T13:35:24.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:23 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:24.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:23 smithi029 ceph-mon[40651]: Deploying daemon mds.b on smithi163 2024-01-24T13:35:24.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:23 smithi029 ceph-mon[40651]: mds.? [v2:172.21.15.29:6840/146067403,v1:172.21.15.29:6841/146067403] up:boot 2024-01-24T13:35:24.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:23 smithi029 ceph-mon[40651]: fsmap 11 up:standby 2024-01-24T13:35:24.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:23 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-01-24T13:35:24.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:24.150+0000 7fd6ad352700 1 -- 172.21.15.29:0/21667989 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6a81a18f0 con 0x7fd6a8100d70 2024-01-24T13:35:24.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:24.150+0000 7fd69ffff700 1 -- 172.21.15.29:0/21667989 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+8843 (secure 0 0 0) 0x7fd690098f50 con 0x7fd6a8100d70 2024-01-24T13:35:24.150 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:35:24.150 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1295649805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1295649805},{"type":"v1","addr":"172.21.15.29:6837","nonce":1295649805}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14559,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2665026288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2665026288},{"type":"v1","addr":"172.21.15.163:6835","nonce":2665026288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":14598,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6839/581075460","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":581075460},{"type":"v1","addr":"172.21.15.29:6839","nonce":581075460}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14610,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6841/146067403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":146067403},{"type":"v1","addr":"172.21.15.29:6841","nonce":146067403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3136190517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3136190517},{"type":"v1","addr":"172.21.15.163:6833","nonce":3136190517}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6833/3688255584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":3688255584},{"type":"v1","addr":"172.21.15.187:6833","nonce":3688255584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24469,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2401960181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2401960181},{"type":"v1","addr":"172.21.15.29:6835","nonce":2401960181}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/1709179808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1709179808},{"type":"v1","addr":"172.21.15.187:6835","nonce":1709179808}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/533724911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":533724911},{"type":"v1","addr":"172.21.15.187:6837","nonce":533724911}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/1610317406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":1610317406},{"type":"v1","addr":"172.21.15.163:6837","nonce":1610317406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24523,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6839/3209015343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6838","nonce":3209015343},{"type":"v1","addr":"172.21.15.187:6839","nonce":3209015343}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2024-01-24T13:35:24.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:24.153+0000 7fd6ad352700 1 -- 172.21.15.29:0/21667989 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fd6940a0570 msgr2=0x7fd6940a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:24.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:24.153+0000 7fd6ad352700 1 --2- 172.21.15.29:0/21667989 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fd6940a0570 0x7fd6940a2a30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd698005950 tx=0x7fd69800b410 comp rx=0 tx=0).stop 2024-01-24T13:35:24.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:24.153+0000 7fd6ad352700 1 -- 172.21.15.29:0/21667989 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6a8100d70 msgr2=0x7fd6a8074af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:24.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:24.153+0000 7fd6ad352700 1 --2- 172.21.15.29:0/21667989 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6a8100d70 0x7fd6a8074af0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd69000e970 tx=0x7fd69000ec80 comp rx=0 tx=0).stop 2024-01-24T13:35:24.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:24.153+0000 7fd6ad352700 1 -- 172.21.15.29:0/21667989 shutdown_connections 2024-01-24T13:35:24.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:24.153+0000 7fd6ad352700 1 --2- 172.21.15.29:0/21667989 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6a8102900 0x7fd6a8073670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:24.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:24.153+0000 7fd6ad352700 1 --2- 172.21.15.29:0/21667989 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fd6940a0570 0x7fd6940a2a30 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:24.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:24.153+0000 7fd6ad352700 1 --2- 172.21.15.29:0/21667989 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6a8100d70 0x7fd6a8074af0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:24.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:24.153+0000 7fd6ad352700 1 --2- 172.21.15.29:0/21667989 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd6a8101f60 0x7fd6a8073130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:24.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:24.153+0000 7fd6ad352700 1 -- 172.21.15.29:0/21667989 >> 172.21.15.29:0/21667989 conn(0x7fd6a80fc2c0 msgr2=0x7fd6a80fe420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:24.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:24.153+0000 7fd6ad352700 1 -- 172.21.15.29:0/21667989 shutdown_connections 2024-01-24T13:35:24.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:24.153+0000 7fd6ad352700 1 -- 172.21.15.29:0/21667989 wait complete. 2024-01-24T13:35:24.154 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 12 2024-01-24T13:35:24.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:23 smithi187 ceph-mon[44911]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:24.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:23 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:24.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:23 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:24.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:23 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:24.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:23 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T13:35:24.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:23 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T13:35:24.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:23 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:24.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:23 smithi187 ceph-mon[44911]: Deploying daemon mds.b on smithi163 2024-01-24T13:35:24.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:23 smithi187 ceph-mon[44911]: mds.? [v2:172.21.15.29:6840/146067403,v1:172.21.15.29:6841/146067403] up:boot 2024-01-24T13:35:24.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:23 smithi187 ceph-mon[44911]: fsmap 11 up:standby 2024-01-24T13:35:24.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:23 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-01-24T13:35:24.319 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jan 24 13:35:24 smithi163 systemd[1]: Starting Ceph mds.b for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:35:24.732 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jan 24 13:35:24 smithi163 ceph-mds[82941]: -- 172.21.15.163:0/3067614259 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558a890231e0 con 0x558a88ff8800 2024-01-24T13:35:24.732 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jan 24 13:35:24 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-b[82918]: starting mds.b at 2024-01-24T13:35:24.732 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jan 24 13:35:24 smithi163 bash[82903]: b328ab6ab77fb41406901373bf836a17d9ce2ecc0866fee8f47a5cc56f6afb7c 2024-01-24T13:35:24.732 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jan 24 13:35:24 smithi163 systemd[1]: Started Ceph mds.b for 83afcf5e-babc-11ee-95b1-87774f69a715. 2024-01-24T13:35:25.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:24 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/21667989' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T13:35:25.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:24 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:25.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:24 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:25.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:24 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:25.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:24 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:25.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:24 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:25.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:24 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:35:25.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:24 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/21667989' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T13:35:25.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:24 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:25.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:24 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:25.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:24 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:25.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:24 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:25.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:24 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:25.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:24 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:35:25.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:24 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/21667989' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T13:35:25.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:24 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:25.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:24 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:25.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:24 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:25.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:24 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:25.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:24 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:25.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:24 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:35:26.167 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T13:35:26.199 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:25 smithi029 ceph-mon[40651]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:26.199 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:25 smithi029 ceph-mon[40651]: mds.? [v2:172.21.15.163:6838/2641650917,v1:172.21.15.163:6839/2641650917] up:boot 2024-01-24T13:35:26.199 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:25 smithi029 ceph-mon[40651]: fsmap 12 up:standby 2024-01-24T13:35:26.199 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:25 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-01-24T13:35:26.199 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:25 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:26.199 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:25 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:26.199 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:25 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:26.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:25 smithi163 ceph-mon[44969]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:26.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:25 smithi163 ceph-mon[44969]: mds.? [v2:172.21.15.163:6838/2641650917,v1:172.21.15.163:6839/2641650917] up:boot 2024-01-24T13:35:26.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:25 smithi163 ceph-mon[44969]: fsmap 12 up:standby 2024-01-24T13:35:26.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:25 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-01-24T13:35:26.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:25 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:26.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:25 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:26.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:25 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:26.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:25 smithi187 ceph-mon[44911]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:26.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:25 smithi187 ceph-mon[44911]: mds.? [v2:172.21.15.163:6838/2641650917,v1:172.21.15.163:6839/2641650917] up:boot 2024-01-24T13:35:26.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:25 smithi187 ceph-mon[44911]: fsmap 12 up:standby 2024-01-24T13:35:26.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:25 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-01-24T13:35:26.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:25 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:26.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:25 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:26.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:25 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:26.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.339+0000 7f3915fd6700 1 -- 172.21.15.29:0/1217541185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39100ffc20 msgr2=0x7f3910100040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:26.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.339+0000 7f3915fd6700 1 --2- 172.21.15.29:0/1217541185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39100ffc20 0x7f3910100040 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f39000099c0 tx=0x7f3900009cd0 comp rx=0 tx=0).stop 2024-01-24T13:35:26.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.339+0000 7f3915fd6700 1 -- 172.21.15.29:0/1217541185 shutdown_connections 2024-01-24T13:35:26.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.339+0000 7f3915fd6700 1 --2- 172.21.15.29:0/1217541185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f39101017b0 0x7f3910108060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:26.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.339+0000 7f3915fd6700 1 --2- 172.21.15.29:0/1217541185 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39100ffc20 0x7f3910100040 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:26.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.339+0000 7f3915fd6700 1 --2- 172.21.15.29:0/1217541185 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3910100e10 0x7f3910101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:26.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.339+0000 7f3915fd6700 1 -- 172.21.15.29:0/1217541185 >> 172.21.15.29:0/1217541185 conn(0x7f39100fb170 msgr2=0x7f39100fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:26.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.339+0000 7f3915fd6700 1 -- 172.21.15.29:0/1217541185 shutdown_connections 2024-01-24T13:35:26.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.339+0000 7f3915fd6700 1 -- 172.21.15.29:0/1217541185 wait complete. 2024-01-24T13:35:26.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.340+0000 7f3915fd6700 1 Processor -- start 2024-01-24T13:35:26.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.340+0000 7f3915fd6700 1 -- start start 2024-01-24T13:35:26.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.340+0000 7f3915fd6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39100ffc20 0x7f3910196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:26.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.340+0000 7f3915fd6700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3910100e10 0x7f3910196f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:26.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.340+0000 7f3915fd6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39101017b0 0x7f391019dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:26.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.340+0000 7f3915fd6700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f391019e580 con 0x7f39100ffc20 2024-01-24T13:35:26.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.340+0000 7f3915fd6700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f391019e6c0 con 0x7f3910100e10 2024-01-24T13:35:26.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.340+0000 7f3915fd6700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f391019e850 con 0x7f39101017b0 2024-01-24T13:35:26.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.340+0000 7f390f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39100ffc20 0x7f3910196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:26.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.340+0000 7f390f7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39100ffc20 0x7f3910196a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59150/0 (socket says 172.21.15.29:59150) 2024-01-24T13:35:26.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.340+0000 7f390f7fe700 1 -- 172.21.15.29:0/189859689 learned_addr learned my addr 172.21.15.29:0/189859689 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:26.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.341+0000 7f390ffff700 1 --2- 172.21.15.29:0/189859689 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39101017b0 0x7f391019dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:26.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.341+0000 7f390effd700 1 --2- 172.21.15.29:0/189859689 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3910100e10 0x7f3910196f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:26.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.341+0000 7f390f7fe700 1 -- 172.21.15.29:0/189859689 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39101017b0 msgr2=0x7f391019dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:26.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.341+0000 7f390f7fe700 1 --2- 172.21.15.29:0/189859689 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39101017b0 0x7f391019dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:26.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.341+0000 7f390f7fe700 1 -- 172.21.15.29:0/189859689 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3910100e10 msgr2=0x7f3910196f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:26.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.341+0000 7f390f7fe700 1 --2- 172.21.15.29:0/189859689 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3910100e10 0x7f3910196f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:26.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.341+0000 7f390f7fe700 1 -- 172.21.15.29:0/189859689 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39000095f0 con 0x7f39100ffc20 2024-01-24T13:35:26.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.341+0000 7f390f7fe700 1 --2- 172.21.15.29:0/189859689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39100ffc20 0x7f3910196a00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3900009990 tx=0x7f390000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:26.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.341+0000 7f390cff9700 1 -- 172.21.15.29:0/189859689 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f390001d070 con 0x7f39100ffc20 2024-01-24T13:35:26.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.341+0000 7f390cff9700 1 -- 172.21.15.29:0/189859689 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3900005030 con 0x7f39100ffc20 2024-01-24T13:35:26.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.341+0000 7f3915fd6700 1 -- 172.21.15.29:0/189859689 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f391019ead0 con 0x7f39100ffc20 2024-01-24T13:35:26.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.341+0000 7f390cff9700 1 -- 172.21.15.29:0/189859689 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3900004b30 con 0x7f39100ffc20 2024-01-24T13:35:26.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.341+0000 7f3915fd6700 1 -- 172.21.15.29:0/189859689 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f391019eef0 con 0x7f39100ffc20 2024-01-24T13:35:26.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.342+0000 7f3915fd6700 1 -- 172.21.15.29:0/189859689 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f391010b480 con 0x7f39100ffc20 2024-01-24T13:35:26.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.343+0000 7f390cff9700 1 -- 172.21.15.29:0/189859689 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f3900005410 con 0x7f39100ffc20 2024-01-24T13:35:26.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.344+0000 7f390cff9700 1 --2- 172.21.15.29:0/189859689 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f38f80a0570 0x7f38f80a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:26.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.344+0000 7f390cff9700 1 -- 172.21.15.29:0/189859689 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f39000cc730 con 0x7f39100ffc20 2024-01-24T13:35:26.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.344+0000 7f390effd700 1 --2- 172.21.15.29:0/189859689 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f38f80a0570 0x7f38f80a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:26.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.345+0000 7f390effd700 1 --2- 172.21.15.29:0/189859689 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f38f80a0570 0x7f38f80a2a30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3904009c60 tx=0x7f3904009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:26.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.348+0000 7f390cff9700 1 -- 172.21.15.29:0/189859689 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f3900096800 con 0x7f39100ffc20 2024-01-24T13:35:26.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.609+0000 7f3915fd6700 1 -- 172.21.15.29:0/189859689 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f391019f4a0 con 0x7f39100ffc20 2024-01-24T13:35:26.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.609+0000 7f390cff9700 1 -- 172.21.15.29:0/189859689 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+9604 (secure 0 0 0) 0x7f3900099e20 con 0x7f39100ffc20 2024-01-24T13:35:26.609 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:35:26.610 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1295649805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1295649805},{"type":"v1","addr":"172.21.15.29:6837","nonce":1295649805}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14559,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2665026288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2665026288},{"type":"v1","addr":"172.21.15.163:6835","nonce":2665026288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":14598,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6839/581075460","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":581075460},{"type":"v1","addr":"172.21.15.29:6839","nonce":581075460}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14610,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6841/146067403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":146067403},{"type":"v1","addr":"172.21.15.29:6841","nonce":146067403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3136190517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3136190517},{"type":"v1","addr":"172.21.15.163:6833","nonce":3136190517}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6833/3688255584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":3688255584},{"type":"v1","addr":"172.21.15.187:6833","nonce":3688255584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24469,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2401960181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2401960181},{"type":"v1","addr":"172.21.15.29:6835","nonce":2401960181}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/1709179808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1709179808},{"type":"v1","addr":"172.21.15.187:6835","nonce":1709179808}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/533724911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":533724911},{"type":"v1","addr":"172.21.15.187:6837","nonce":533724911}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/1610317406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":1610317406},{"type":"v1","addr":"172.21.15.163:6837","nonce":1610317406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24523,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6839/3209015343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6838","nonce":3209015343},{"type":"v1","addr":"172.21.15.187:6839","nonce":3209015343}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24548,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6839/2641650917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":2641650917},{"type":"v1","addr":"172.21.15.163:6839","nonce":2641650917}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[]} 2024-01-24T13:35:26.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.612+0000 7f3915fd6700 1 -- 172.21.15.29:0/189859689 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f38f80a0570 msgr2=0x7f38f80a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:26.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.612+0000 7f3915fd6700 1 --2- 172.21.15.29:0/189859689 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f38f80a0570 0x7f38f80a2a30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3904009c60 tx=0x7f3904009260 comp rx=0 tx=0).stop 2024-01-24T13:35:26.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.612+0000 7f3915fd6700 1 -- 172.21.15.29:0/189859689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39100ffc20 msgr2=0x7f3910196a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:26.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.612+0000 7f3915fd6700 1 --2- 172.21.15.29:0/189859689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39100ffc20 0x7f3910196a00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3900009990 tx=0x7f390000f690 comp rx=0 tx=0).stop 2024-01-24T13:35:26.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.612+0000 7f3915fd6700 1 -- 172.21.15.29:0/189859689 shutdown_connections 2024-01-24T13:35:26.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.612+0000 7f3915fd6700 1 --2- 172.21.15.29:0/189859689 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39101017b0 0x7f391019dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:26.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.612+0000 7f3915fd6700 1 --2- 172.21.15.29:0/189859689 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f38f80a0570 0x7f38f80a2a30 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:26.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.612+0000 7f3915fd6700 1 --2- 172.21.15.29:0/189859689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f39100ffc20 0x7f3910196a00 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:26.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.612+0000 7f3915fd6700 1 --2- 172.21.15.29:0/189859689 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3910100e10 0x7f3910196f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:26.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.612+0000 7f3915fd6700 1 -- 172.21.15.29:0/189859689 >> 172.21.15.29:0/189859689 conn(0x7f39100fb170 msgr2=0x7f3910067ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:26.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.612+0000 7f3915fd6700 1 -- 172.21.15.29:0/189859689 shutdown_connections 2024-01-24T13:35:26.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.612+0000 7f3915fd6700 1 -- 172.21.15.29:0/189859689 wait complete. 2024-01-24T13:35:26.613 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 13 2024-01-24T13:35:26.626 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T13:35:26.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.817+0000 7fe9c644b700 1 -- 172.21.15.29:0/3924483225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9c0104100 msgr2=0x7fe9c01064f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:26.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.817+0000 7fe9c644b700 1 --2- 172.21.15.29:0/3924483225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9c0104100 0x7fe9c01064f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4009990 tx=0x7fe9b4009ca0 comp rx=0 tx=0).stop 2024-01-24T13:35:26.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.817+0000 7fe9c644b700 1 -- 172.21.15.29:0/3924483225 shutdown_connections 2024-01-24T13:35:26.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.817+0000 7fe9c644b700 1 --2- 172.21.15.29:0/3924483225 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9c0106a30 0x7fe9c0108ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:26.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.817+0000 7fe9c644b700 1 --2- 172.21.15.29:0/3924483225 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe9c01017d0 0x7fe9c0103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:26.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.817+0000 7fe9c644b700 1 --2- 172.21.15.29:0/3924483225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9c0104100 0x7fe9c01064f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:26.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.817+0000 7fe9c644b700 1 -- 172.21.15.29:0/3924483225 >> 172.21.15.29:0/3924483225 conn(0x7fe9c00fb1a0 msgr2=0x7fe9c00fd600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:26.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.817+0000 7fe9c644b700 1 -- 172.21.15.29:0/3924483225 shutdown_connections 2024-01-24T13:35:26.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.817+0000 7fe9c644b700 1 -- 172.21.15.29:0/3924483225 wait complete. 2024-01-24T13:35:26.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.818+0000 7fe9c644b700 1 Processor -- start 2024-01-24T13:35:26.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.818+0000 7fe9c644b700 1 -- start start 2024-01-24T13:35:26.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.818+0000 7fe9c644b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe9c01017d0 0x7fe9c01947a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:26.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.818+0000 7fe9c644b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9c0104100 0x7fe9c0194ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:26.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.818+0000 7fe9c644b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9c0106a30 0x7fe9c019bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:26.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.818+0000 7fe9c644b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c019c320 con 0x7fe9c0106a30 2024-01-24T13:35:26.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.818+0000 7fe9c644b700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c019c460 con 0x7fe9c01017d0 2024-01-24T13:35:26.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.818+0000 7fe9c644b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c019c5f0 con 0x7fe9c0104100 2024-01-24T13:35:26.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.818+0000 7fe9bf7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9c0104100 0x7fe9c0194ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:26.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.818+0000 7fe9bffff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe9c01017d0 0x7fe9c01947a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:26.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.818+0000 7fe9bf7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9c0104100 0x7fe9c0194ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.29:57522/0 (socket says 172.21.15.29:57522) 2024-01-24T13:35:26.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.818+0000 7fe9bf7fe700 1 -- 172.21.15.29:0/1401677170 learned_addr learned my addr 172.21.15.29:0/1401677170 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:26.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.818+0000 7fe9c49e8700 1 --2- 172.21.15.29:0/1401677170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9c0106a30 0x7fe9c019bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:26.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.819+0000 7fe9bffff700 1 -- 172.21.15.29:0/1401677170 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9c0104100 msgr2=0x7fe9c0194ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:26.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.819+0000 7fe9bffff700 1 --2- 172.21.15.29:0/1401677170 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9c0104100 0x7fe9c0194ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:26.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.819+0000 7fe9bffff700 1 -- 172.21.15.29:0/1401677170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9c0106a30 msgr2=0x7fe9c019bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:26.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.819+0000 7fe9bffff700 1 --2- 172.21.15.29:0/1401677170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9c0106a30 0x7fe9c019bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:26.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.819+0000 7fe9bffff700 1 -- 172.21.15.29:0/1401677170 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b40095f0 con 0x7fe9c01017d0 2024-01-24T13:35:26.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.819+0000 7fe9bf7fe700 1 --2- 172.21.15.29:0/1401677170 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9c0104100 0x7fe9c0194ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T13:35:26.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.819+0000 7fe9bffff700 1 --2- 172.21.15.29:0/1401677170 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe9c01017d0 0x7fe9c01947a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0009820 tx=0x7fe9b0009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:26.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.819+0000 7fe9bd7fa700 1 -- 172.21.15.29:0/1401677170 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b000c940 con 0x7fe9c01017d0 2024-01-24T13:35:26.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.819+0000 7fe9bd7fa700 1 -- 172.21.15.29:0/1401677170 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe9b0017400 con 0x7fe9c01017d0 2024-01-24T13:35:26.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.819+0000 7fe9bd7fa700 1 -- 172.21.15.29:0/1401677170 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b00165c0 con 0x7fe9c01017d0 2024-01-24T13:35:26.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.819+0000 7fe9c644b700 1 -- 172.21.15.29:0/1401677170 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9c019c8d0 con 0x7fe9c01017d0 2024-01-24T13:35:26.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.819+0000 7fe9c644b700 1 -- 172.21.15.29:0/1401677170 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9c019cde0 con 0x7fe9c01017d0 2024-01-24T13:35:26.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.820+0000 7fe9c644b700 1 -- 172.21.15.29:0/1401677170 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9c010a3f0 con 0x7fe9c01017d0 2024-01-24T13:35:26.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.822+0000 7fe9bd7fa700 1 -- 172.21.15.29:0/1401677170 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7fe9b0017870 con 0x7fe9c01017d0 2024-01-24T13:35:26.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.822+0000 7fe9bd7fa700 1 --2- 172.21.15.29:0/1401677170 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fe9a80a05f0 0x7fe9a80a2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:26.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.823+0000 7fe9bd7fa700 1 -- 172.21.15.29:0/1401677170 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe9b00cb270 con 0x7fe9c01017d0 2024-01-24T13:35:26.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.823+0000 7fe9bf7fe700 1 --2- 172.21.15.29:0/1401677170 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fe9a80a05f0 0x7fe9a80a2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:26.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.823+0000 7fe9bf7fe700 1 --2- 172.21.15.29:0/1401677170 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fe9a80a05f0 0x7fe9a80a2ab0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4016fd0 tx=0x7fe9b4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:26.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:26.826+0000 7fe9bd7fa700 1 -- 172.21.15.29:0/1401677170 <== mon.2 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fe9b0095340 con 0x7fe9c01017d0 2024-01-24T13:35:27.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.082+0000 7fe9c644b700 1 -- 172.21.15.29:0/1401677170 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9c019d420 con 0x7fe9c01017d0 2024-01-24T13:35:27.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.084+0000 7fe9bd7fa700 1 -- 172.21.15.29:0/1401677170 <== mon.2 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+18443 (secure 0 0 0) 0x7fe9b0098960 con 0x7fe9c01017d0 2024-01-24T13:35:27.084 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:35:27.084 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":77,"fsid":"83afcf5e-babc-11ee-95b1-87774f69a715","created":"2024-01-24T13:29:33.175895+0000","modified":"2024-01-24T13:34:58.148429+0000","last_up_change":"2024-01-24T13:34:57.141902+0000","last_in_change":"2024-01-24T13:34:41.071943+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T13:32:21.264634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9857a607-2e6f-41a5-a86a-795970be6210","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":2406629076},{"type":"v1","addr":"172.21.15.29:6803","nonce":2406629076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":2406629076},{"type":"v1","addr":"172.21.15.29:6805","nonce":2406629076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2406629076},{"type":"v1","addr":"172.21.15.29:6809","nonce":2406629076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":2406629076},{"type":"v1","addr":"172.21.15.29:6807","nonce":2406629076}]},"public_addr":"172.21.15.29:6803/2406629076","cluster_addr":"172.21.15.29:6805/2406629076","heartbeat_back_addr":"172.21.15.29:6809/2406629076","heartbeat_front_addr":"172.21.15.29:6807/2406629076","state":["exists","up"]},{"osd":1,"uuid":"8bb2ba48-23d4-4bbd-a912-ae74f9e7f6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":2217244566},{"type":"v1","addr":"172.21.15.163:6801","nonce":2217244566}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":2217244566},{"type":"v1","addr":"172.21.15.163:6803","nonce":2217244566}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":2217244566},{"type":"v1","addr":"172.21.15.163:6807","nonce":2217244566}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":2217244566},{"type":"v1","addr":"172.21.15.163:6805","nonce":2217244566}]},"public_addr":"172.21.15.163:6801/2217244566","cluster_addr":"172.21.15.163:6803/2217244566","heartbeat_back_addr":"172.21.15.163:6807/2217244566","heartbeat_front_addr":"172.21.15.163:6805/2217244566","state":["exists","up"]},{"osd":2,"uuid":"f0ed4131-cd29-4706-89ea-fde40f46ea83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3589694992},{"type":"v1","addr":"172.21.15.187:6801","nonce":3589694992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3589694992},{"type":"v1","addr":"172.21.15.187:6803","nonce":3589694992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3589694992},{"type":"v1","addr":"172.21.15.187:6807","nonce":3589694992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3589694992},{"type":"v1","addr":"172.21.15.187:6805","nonce":3589694992}]},"public_addr":"172.21.15.187:6801/3589694992","cluster_addr":"172.21.15.187:6803/3589694992","heartbeat_back_addr":"172.21.15.187:6807/3589694992","heartbeat_front_addr":"172.21.15.187:6805/3589694992","state":["exists","up"]},{"osd":3,"uuid":"da821310-b502-4b77-a5ce-0ffed0a3dafc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3417210415},{"type":"v1","addr":"172.21.15.29:6811","nonce":3417210415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":3417210415},{"type":"v1","addr":"172.21.15.29:6813","nonce":3417210415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3417210415},{"type":"v1","addr":"172.21.15.29:6817","nonce":3417210415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3417210415},{"type":"v1","addr":"172.21.15.29:6815","nonce":3417210415}]},"public_addr":"172.21.15.29:6811/3417210415","cluster_addr":"172.21.15.29:6813/3417210415","heartbeat_back_addr":"172.21.15.29:6817/3417210415","heartbeat_front_addr":"172.21.15.29:6815/3417210415","state":["exists","up"]},{"osd":4,"uuid":"8b3a348f-e7dd-4ccb-82a5-2d3f5d101da8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3283787183},{"type":"v1","addr":"172.21.15.163:6809","nonce":3283787183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":3283787183},{"type":"v1","addr":"172.21.15.163:6811","nonce":3283787183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3283787183},{"type":"v1","addr":"172.21.15.163:6815","nonce":3283787183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":3283787183},{"type":"v1","addr":"172.21.15.163:6813","nonce":3283787183}]},"public_addr":"172.21.15.163:6809/3283787183","cluster_addr":"172.21.15.163:6811/3283787183","heartbeat_back_addr":"172.21.15.163:6815/3283787183","heartbeat_front_addr":"172.21.15.163:6813/3283787183","state":["exists","up"]},{"osd":5,"uuid":"c26c7155-618b-4712-acdb-d4fd6c8b7f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":2259228817},{"type":"v1","addr":"172.21.15.187:6809","nonce":2259228817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":2259228817},{"type":"v1","addr":"172.21.15.187:6811","nonce":2259228817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":2259228817},{"type":"v1","addr":"172.21.15.187:6815","nonce":2259228817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":2259228817},{"type":"v1","addr":"172.21.15.187:6813","nonce":2259228817}]},"public_addr":"172.21.15.187:6809/2259228817","cluster_addr":"172.21.15.187:6811/2259228817","heartbeat_back_addr":"172.21.15.187:6815/2259228817","heartbeat_front_addr":"172.21.15.187:6813/2259228817","state":["exists","up"]},{"osd":6,"uuid":"042ac0b4-1fe1-444f-bf51-ad89613a2185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":938400701},{"type":"v1","addr":"172.21.15.29:6819","nonce":938400701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":938400701},{"type":"v1","addr":"172.21.15.29:6821","nonce":938400701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":938400701},{"type":"v1","addr":"172.21.15.29:6825","nonce":938400701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":938400701},{"type":"v1","addr":"172.21.15.29:6823","nonce":938400701}]},"public_addr":"172.21.15.29:6819/938400701","cluster_addr":"172.21.15.29:6821/938400701","heartbeat_back_addr":"172.21.15.29:6825/938400701","heartbeat_front_addr":"172.21.15.29:6823/938400701","state":["exists","up"]},{"osd":7,"uuid":"add3d09b-76f3-4a93-83a2-30ac94c3fad1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2783976092},{"type":"v1","addr":"172.21.15.163:6817","nonce":2783976092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2783976092},{"type":"v1","addr":"172.21.15.163:6819","nonce":2783976092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2783976092},{"type":"v1","addr":"172.21.15.163:6823","nonce":2783976092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2783976092},{"type":"v1","addr":"172.21.15.163:6821","nonce":2783976092}]},"public_addr":"172.21.15.163:6817/2783976092","cluster_addr":"172.21.15.163:6819/2783976092","heartbeat_back_addr":"172.21.15.163:6823/2783976092","heartbeat_front_addr":"172.21.15.163:6821/2783976092","state":["exists","up"]},{"osd":8,"uuid":"71044d6a-b55f-4bcf-94c5-79f71eb7b422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":2856803882},{"type":"v1","addr":"172.21.15.187:6817","nonce":2856803882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":2856803882},{"type":"v1","addr":"172.21.15.187:6819","nonce":2856803882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":2856803882},{"type":"v1","addr":"172.21.15.187:6823","nonce":2856803882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":2856803882},{"type":"v1","addr":"172.21.15.187:6821","nonce":2856803882}]},"public_addr":"172.21.15.187:6817/2856803882","cluster_addr":"172.21.15.187:6819/2856803882","heartbeat_back_addr":"172.21.15.187:6823/2856803882","heartbeat_front_addr":"172.21.15.187:6821/2856803882","state":["exists","up"]},{"osd":9,"uuid":"cf417a8e-e576-43de-b89c-f104ee3969ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1392813017},{"type":"v1","addr":"172.21.15.29:6827","nonce":1392813017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1392813017},{"type":"v1","addr":"172.21.15.29:6829","nonce":1392813017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1392813017},{"type":"v1","addr":"172.21.15.29:6833","nonce":1392813017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1392813017},{"type":"v1","addr":"172.21.15.29:6831","nonce":1392813017}]},"public_addr":"172.21.15.29:6827/1392813017","cluster_addr":"172.21.15.29:6829/1392813017","heartbeat_back_addr":"172.21.15.29:6833/1392813017","heartbeat_front_addr":"172.21.15.29:6831/1392813017","state":["exists","up"]},{"osd":10,"uuid":"bc7cfb9d-3bfa-45b2-929a-a8dc90f1a2a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":2523735721},{"type":"v1","addr":"172.21.15.163:6825","nonce":2523735721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":2523735721},{"type":"v1","addr":"172.21.15.163:6827","nonce":2523735721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":2523735721},{"type":"v1","addr":"172.21.15.163:6831","nonce":2523735721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":2523735721},{"type":"v1","addr":"172.21.15.163:6829","nonce":2523735721}]},"public_addr":"172.21.15.163:6825/2523735721","cluster_addr":"172.21.15.163:6827/2523735721","heartbeat_back_addr":"172.21.15.163:6831/2523735721","heartbeat_front_addr":"172.21.15.163:6829/2523735721","state":["exists","up"]},{"osd":11,"uuid":"1453684e-fe11-4cbc-a868-cf90b3a7a021","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1828122314},{"type":"v1","addr":"172.21.15.187:6825","nonce":1828122314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1828122314},{"type":"v1","addr":"172.21.15.187:6827","nonce":1828122314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":1828122314},{"type":"v1","addr":"172.21.15.187:6831","nonce":1828122314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":1828122314},{"type":"v1","addr":"172.21.15.187:6829","nonce":1828122314}]},"public_addr":"172.21.15.187:6825/1828122314","cluster_addr":"172.21.15.187:6827/1828122314","heartbeat_back_addr":"172.21.15.187:6831/1828122314","heartbeat_front_addr":"172.21.15.187:6829/1828122314","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:31:43.572543+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:32:00.759082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:32:17.705002+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:32:35.666788+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:32:53.221701+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:33:10.290988+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:33:27.946743+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:33:45.698074+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:34:02.750761+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:34:20.662278+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:34:38.233389+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:34:55.419540+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/1419474286":"2024-01-25T13:30:30.728638+0000","172.21.15.29:6800/835547951":"2024-01-25T13:30:08.775587+0000","172.21.15.29:6801/1996568496":"2024-01-25T13:30:30.728638+0000","172.21.15.29:0/2727527401":"2024-01-25T13:30:08.775587+0000","172.21.15.29:0/41509684":"2024-01-25T13:30:30.728638+0000","172.21.15.29:6800/1996568496":"2024-01-25T13:30:30.728638+0000","172.21.15.29:0/2833864599":"2024-01-25T13:30:30.728638+0000","172.21.15.29:0/168349482":"2024-01-25T13:30:08.775587+0000","172.21.15.29:6801/835547951":"2024-01-25T13:30:08.775587+0000","172.21.15.29:0/1142561996":"2024-01-25T13:30:08.775587+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T13:35:27.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.086+0000 7fe9c644b700 1 -- 172.21.15.29:0/1401677170 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fe9a80a05f0 msgr2=0x7fe9a80a2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:27.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.086+0000 7fe9c644b700 1 --2- 172.21.15.29:0/1401677170 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fe9a80a05f0 0x7fe9a80a2ab0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4016fd0 tx=0x7fe9b4005fb0 comp rx=0 tx=0).stop 2024-01-24T13:35:27.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.086+0000 7fe9c644b700 1 -- 172.21.15.29:0/1401677170 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe9c01017d0 msgr2=0x7fe9c01947a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:27.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.086+0000 7fe9c644b700 1 --2- 172.21.15.29:0/1401677170 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe9c01017d0 0x7fe9c01947a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0009820 tx=0x7fe9b0009b30 comp rx=0 tx=0).stop 2024-01-24T13:35:27.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.086+0000 7fe9c644b700 1 -- 172.21.15.29:0/1401677170 shutdown_connections 2024-01-24T13:35:27.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.086+0000 7fe9c644b700 1 --2- 172.21.15.29:0/1401677170 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fe9a80a05f0 0x7fe9a80a2ab0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:27.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.086+0000 7fe9c644b700 1 --2- 172.21.15.29:0/1401677170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9c0106a30 0x7fe9c019bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:27.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.086+0000 7fe9c644b700 1 --2- 172.21.15.29:0/1401677170 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe9c01017d0 0x7fe9c01947a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:27.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.086+0000 7fe9c644b700 1 --2- 172.21.15.29:0/1401677170 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9c0104100 0x7fe9c0194ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:27.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.086+0000 7fe9c644b700 1 -- 172.21.15.29:0/1401677170 >> 172.21.15.29:0/1401677170 conn(0x7fe9c00fb1a0 msgr2=0x7fe9c0102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:27.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.086+0000 7fe9c644b700 1 -- 172.21.15.29:0/1401677170 shutdown_connections 2024-01-24T13:35:27.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.086+0000 7fe9c644b700 1 -- 172.21.15.29:0/1401677170 wait complete. 2024-01-24T13:35:27.100 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T13:32:21.264634+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T13:35:27.100 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T13:35:27.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:26 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:27.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:26 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:27.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:26 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:27.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:26 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:27.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:26 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:27.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:26 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:35:27.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:26 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:27.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:26 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:35:27.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:26 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/189859689' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T13:35:27.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.244+0000 7fc9215e3700 1 -- 172.21.15.29:0/164406694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc91c100e90 msgr2=0x7fc91c1012f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:27.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.244+0000 7fc9215e3700 1 --2- 172.21.15.29:0/164406694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc91c100e90 0x7fc91c1012f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc9100099c0 tx=0x7fc910009cd0 comp rx=0 tx=0).stop 2024-01-24T13:35:27.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.244+0000 7fc9215e3700 1 -- 172.21.15.29:0/164406694 shutdown_connections 2024-01-24T13:35:27.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.244+0000 7fc9215e3700 1 --2- 172.21.15.29:0/164406694 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc91c101830 0x7fc91c1080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:27.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.244+0000 7fc9215e3700 1 --2- 172.21.15.29:0/164406694 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc91c0ffca0 0x7fc91c1000c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:27.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.244+0000 7fc9215e3700 1 --2- 172.21.15.29:0/164406694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc91c100e90 0x7fc91c1012f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:27.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.244+0000 7fc9215e3700 1 -- 172.21.15.29:0/164406694 >> 172.21.15.29:0/164406694 conn(0x7fc91c0fb210 msgr2=0x7fc91c0fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:27.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.244+0000 7fc9215e3700 1 -- 172.21.15.29:0/164406694 shutdown_connections 2024-01-24T13:35:27.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.244+0000 7fc9215e3700 1 -- 172.21.15.29:0/164406694 wait complete. 2024-01-24T13:35:27.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.245+0000 7fc9215e3700 1 Processor -- start 2024-01-24T13:35:27.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.245+0000 7fc9215e3700 1 -- start start 2024-01-24T13:35:27.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.245+0000 7fc9215e3700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc91c0ffca0 0x7fc91c198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:27.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.245+0000 7fc9215e3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc91c100e90 0x7fc91c199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:27.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.245+0000 7fc9215e3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc91c101830 0x7fc91c1a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:27.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.245+0000 7fc9215e3700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc91c1a0770 con 0x7fc91c101830 2024-01-24T13:35:27.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.245+0000 7fc9215e3700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc91c1a08b0 con 0x7fc91c0ffca0 2024-01-24T13:35:27.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.245+0000 7fc9215e3700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc91c1a0a40 con 0x7fc91c100e90 2024-01-24T13:35:27.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.245+0000 7fc91b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc91c101830 0x7fc91c1a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:27.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.245+0000 7fc91a7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc91c100e90 0x7fc91c199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:27.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.245+0000 7fc91b7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc91c101830 0x7fc91c1a01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59180/0 (socket says 172.21.15.29:59180) 2024-01-24T13:35:27.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.245+0000 7fc91b7fe700 1 -- 172.21.15.29:0/885656037 learned_addr learned my addr 172.21.15.29:0/885656037 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:27.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.245+0000 7fc91affd700 1 --2- 172.21.15.29:0/885656037 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc91c0ffca0 0x7fc91c198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:27.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.246+0000 7fc91b7fe700 1 -- 172.21.15.29:0/885656037 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc91c100e90 msgr2=0x7fc91c199130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:27.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.246+0000 7fc91b7fe700 1 --2- 172.21.15.29:0/885656037 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc91c100e90 0x7fc91c199130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:27.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.246+0000 7fc91b7fe700 1 -- 172.21.15.29:0/885656037 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc91c0ffca0 msgr2=0x7fc91c198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:27.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.246+0000 7fc91b7fe700 1 --2- 172.21.15.29:0/885656037 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc91c0ffca0 0x7fc91c198bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:27.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.246+0000 7fc91b7fe700 1 -- 172.21.15.29:0/885656037 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9100095f0 con 0x7fc91c101830 2024-01-24T13:35:27.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.246+0000 7fc91a7fc700 1 --2- 172.21.15.29:0/885656037 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc91c100e90 0x7fc91c199130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T13:35:27.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.246+0000 7fc91b7fe700 1 --2- 172.21.15.29:0/885656037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc91c101830 0x7fc91c1a01a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc91400ef30 tx=0x7fc91400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:27.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.246+0000 7fc903fff700 1 -- 172.21.15.29:0/885656037 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc914018070 con 0x7fc91c101830 2024-01-24T13:35:27.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.246+0000 7fc9215e3700 1 -- 172.21.15.29:0/885656037 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc91c1a0cc0 con 0x7fc91c101830 2024-01-24T13:35:27.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.246+0000 7fc903fff700 1 -- 172.21.15.29:0/885656037 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc91400f040 con 0x7fc91c101830 2024-01-24T13:35:27.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.246+0000 7fc903fff700 1 -- 172.21.15.29:0/885656037 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91400cd00 con 0x7fc91c101830 2024-01-24T13:35:27.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.246+0000 7fc9215e3700 1 -- 172.21.15.29:0/885656037 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc91c1a1180 con 0x7fc91c101830 2024-01-24T13:35:27.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.247+0000 7fc9215e3700 1 -- 172.21.15.29:0/885656037 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc91c10b440 con 0x7fc91c101830 2024-01-24T13:35:27.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.248+0000 7fc903fff700 1 -- 172.21.15.29:0/885656037 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7fc914003a00 con 0x7fc91c101830 2024-01-24T13:35:27.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.249+0000 7fc903fff700 1 --2- 172.21.15.29:0/885656037 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fc9040a0520 0x7fc9040a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:27.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.249+0000 7fc903fff700 1 -- 172.21.15.29:0/885656037 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc9140cb050 con 0x7fc91c101830 2024-01-24T13:35:27.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.249+0000 7fc91affd700 1 --2- 172.21.15.29:0/885656037 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fc9040a0520 0x7fc9040a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:27.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.249+0000 7fc91affd700 1 --2- 172.21.15.29:0/885656037 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fc9040a0520 0x7fc9040a29e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc91c100cf0 tx=0x7fc90c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:27.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.253+0000 7fc903fff700 1 -- 172.21.15.29:0/885656037 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fc914095120 con 0x7fc91c101830 2024-01-24T13:35:27.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:26 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:27.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:26 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:27.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:26 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:27.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:26 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:27.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:26 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:27.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:26 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:35:27.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:26 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:27.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:26 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:35:27.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:26 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/189859689' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T13:35:27.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:26 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:27.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:26 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:27.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:26 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:27.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:26 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:27.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:26 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:27.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:26 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:35:27.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:26 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:27.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:26 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T13:35:27.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:26 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/189859689' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T13:35:27.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.505+0000 7fc9215e3700 1 -- 172.21.15.29:0/885656037 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc91c1a1730 con 0x7fc91c101830 2024-01-24T13:35:27.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.505+0000 7fc903fff700 1 -- 172.21.15.29:0/885656037 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc914098740 con 0x7fc91c101830 2024-01-24T13:35:27.505 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-01-24T13:35:27.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.507+0000 7fc9215e3700 1 -- 172.21.15.29:0/885656037 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fc9040a0520 msgr2=0x7fc9040a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:27.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.507+0000 7fc9215e3700 1 --2- 172.21.15.29:0/885656037 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fc9040a0520 0x7fc9040a29e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc91c100cf0 tx=0x7fc90c009260 comp rx=0 tx=0).stop 2024-01-24T13:35:27.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.508+0000 7fc9215e3700 1 -- 172.21.15.29:0/885656037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc91c101830 msgr2=0x7fc91c1a01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:27.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.508+0000 7fc9215e3700 1 --2- 172.21.15.29:0/885656037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc91c101830 0x7fc91c1a01a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc91400ef30 tx=0x7fc91400c3a0 comp rx=0 tx=0).stop 2024-01-24T13:35:27.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.508+0000 7fc9215e3700 1 -- 172.21.15.29:0/885656037 shutdown_connections 2024-01-24T13:35:27.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.508+0000 7fc9215e3700 1 --2- 172.21.15.29:0/885656037 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fc9040a0520 0x7fc9040a29e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:27.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.508+0000 7fc9215e3700 1 --2- 172.21.15.29:0/885656037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc91c101830 0x7fc91c1a01a0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:27.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.508+0000 7fc9215e3700 1 --2- 172.21.15.29:0/885656037 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc91c0ffca0 0x7fc91c198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:27.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.508+0000 7fc9215e3700 1 --2- 172.21.15.29:0/885656037 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc91c100e90 0x7fc91c199130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:27.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.508+0000 7fc9215e3700 1 -- 172.21.15.29:0/885656037 >> 172.21.15.29:0/885656037 conn(0x7fc91c0fb210 msgr2=0x7fc91c069110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:27.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.509+0000 7fc9215e3700 1 -- 172.21.15.29:0/885656037 shutdown_connections 2024-01-24T13:35:27.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.509+0000 7fc9215e3700 1 -- 172.21.15.29:0/885656037 wait complete. 2024-01-24T13:35:27.524 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-24T13:35:27.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.672+0000 7f43cb66a700 1 -- 172.21.15.29:0/3005886921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43c4101210 msgr2=0x7f43c4101630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:27.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.672+0000 7f43cb66a700 1 --2- 172.21.15.29:0/3005886921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43c4101210 0x7f43c4101630 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f43c0009990 tx=0x7f43c0009ca0 comp rx=0 tx=0).stop 2024-01-24T13:35:27.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.672+0000 7f43cb66a700 1 -- 172.21.15.29:0/3005886921 shutdown_connections 2024-01-24T13:35:27.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.672+0000 7f43cb66a700 1 --2- 172.21.15.29:0/3005886921 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43c4100850 0x7f43c4108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:27.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.672+0000 7f43cb66a700 1 --2- 172.21.15.29:0/3005886921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43c4101210 0x7f43c4101630 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:27.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.672+0000 7f43cb66a700 1 --2- 172.21.15.29:0/3005886921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43c40ffeb0 0x7f43c4100310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:27.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.672+0000 7f43cb66a700 1 -- 172.21.15.29:0/3005886921 >> 172.21.15.29:0/3005886921 conn(0x7f43c40fa900 msgr2=0x7f43c40fcd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:27.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.672+0000 7f43cb66a700 1 -- 172.21.15.29:0/3005886921 shutdown_connections 2024-01-24T13:35:27.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.672+0000 7f43cb66a700 1 -- 172.21.15.29:0/3005886921 wait complete. 2024-01-24T13:35:27.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.672+0000 7f43cb66a700 1 Processor -- start 2024-01-24T13:35:27.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.672+0000 7f43cb66a700 1 -- start start 2024-01-24T13:35:27.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.673+0000 7f43cb66a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43c40ffeb0 0x7f43c4196a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:27.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.673+0000 7f43cb66a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43c4100850 0x7f43c4196f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:27.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.673+0000 7f43cb66a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43c4101210 0x7f43c419dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:27.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.673+0000 7f43cb66a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43c419e5a0 con 0x7f43c4101210 2024-01-24T13:35:27.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.673+0000 7f43cb66a700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43c419e6e0 con 0x7f43c4100850 2024-01-24T13:35:27.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.673+0000 7f43cb66a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43c419e870 con 0x7f43c40ffeb0 2024-01-24T13:35:27.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.673+0000 7f43c9c07700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43c4101210 0x7f43c419dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:27.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.673+0000 7f43c9c07700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43c4101210 0x7f43c419dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59194/0 (socket says 172.21.15.29:59194) 2024-01-24T13:35:27.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.673+0000 7f43c9c07700 1 -- 172.21.15.29:0/3009956260 learned_addr learned my addr 172.21.15.29:0/3009956260 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:27.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.673+0000 7f43c9406700 1 --2- 172.21.15.29:0/3009956260 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43c40ffeb0 0x7f43c4196a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:27.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.673+0000 7f43c8c05700 1 --2- 172.21.15.29:0/3009956260 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43c4100850 0x7f43c4196f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:27.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.673+0000 7f43c9c07700 1 -- 172.21.15.29:0/3009956260 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43c40ffeb0 msgr2=0x7f43c4196a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:27.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.673+0000 7f43c9c07700 1 --2- 172.21.15.29:0/3009956260 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43c40ffeb0 0x7f43c4196a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:27.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.673+0000 7f43c9c07700 1 -- 172.21.15.29:0/3009956260 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43c4100850 msgr2=0x7f43c4196f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:27.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.673+0000 7f43c9c07700 1 --2- 172.21.15.29:0/3009956260 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43c4100850 0x7f43c4196f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:27.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.673+0000 7f43c9c07700 1 -- 172.21.15.29:0/3009956260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43c00095f0 con 0x7f43c4101210 2024-01-24T13:35:27.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.673+0000 7f43c9406700 1 --2- 172.21.15.29:0/3009956260 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43c40ffeb0 0x7f43c4196a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T13:35:27.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.674+0000 7f43c9c07700 1 --2- 172.21.15.29:0/3009956260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43c4101210 0x7f43c419dfd0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f43bc00ef30 tx=0x7f43bc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:27.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.674+0000 7f43ba7fc700 1 -- 172.21.15.29:0/3009956260 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43bc018070 con 0x7f43c4101210 2024-01-24T13:35:27.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.674+0000 7f43ba7fc700 1 -- 172.21.15.29:0/3009956260 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f43bc00f040 con 0x7f43c4101210 2024-01-24T13:35:27.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.674+0000 7f43cb66a700 1 -- 172.21.15.29:0/3009956260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43c419eaf0 con 0x7f43c4101210 2024-01-24T13:35:27.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.674+0000 7f43ba7fc700 1 -- 172.21.15.29:0/3009956260 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43bc00cd00 con 0x7f43c4101210 2024-01-24T13:35:27.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.674+0000 7f43cb66a700 1 -- 172.21.15.29:0/3009956260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43c419f000 con 0x7f43c4101210 2024-01-24T13:35:27.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.675+0000 7f43cb66a700 1 -- 172.21.15.29:0/3009956260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43c410b480 con 0x7f43c4101210 2024-01-24T13:35:27.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.676+0000 7f43ba7fc700 1 -- 172.21.15.29:0/3009956260 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f43bc003a00 con 0x7f43c4101210 2024-01-24T13:35:27.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.677+0000 7f43ba7fc700 1 --2- 172.21.15.29:0/3009956260 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f43b00a05f0 0x7f43b00a2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:27.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.677+0000 7f43c9406700 1 --2- 172.21.15.29:0/3009956260 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f43b00a05f0 0x7f43b00a2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:27.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.677+0000 7f43ba7fc700 1 -- 172.21.15.29:0/3009956260 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f43bc0cafc0 con 0x7f43c4101210 2024-01-24T13:35:27.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.678+0000 7f43c9406700 1 --2- 172.21.15.29:0/3009956260 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f43b00a05f0 0x7f43b00a2ab0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f43c000b2e0 tx=0x7f43c0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:27.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.682+0000 7f43ba7fc700 1 -- 172.21.15.29:0/3009956260 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f43bc0fe120 con 0x7f43c4101210 2024-01-24T13:35:27.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.928+0000 7f43cb66a700 1 -- 172.21.15.29:0/3009956260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f43c410b6a0 con 0x7f43c4101210 2024-01-24T13:35:27.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.928+0000 7f43ba7fc700 1 -- 172.21.15.29:0/3009956260 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v77) v1 ==== 84+0+65 (secure 0 0 0) 0x7f43bc095090 con 0x7f43c4101210 2024-01-24T13:35:27.928 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:35:27.928 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2024-01-24T13:35:27.928 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-01-24T13:35:27.928 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2024-01-24T13:35:27.928 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2024-01-24T13:35:27.928 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-01-24T13:35:27.928 INFO:teuthology.orchestra.run.smithi029.stdout:] 2024-01-24T13:35:27.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.931+0000 7f43cb66a700 1 -- 172.21.15.29:0/3009956260 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f43b00a05f0 msgr2=0x7f43b00a2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:27.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.931+0000 7f43cb66a700 1 --2- 172.21.15.29:0/3009956260 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f43b00a05f0 0x7f43b00a2ab0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f43c000b2e0 tx=0x7f43c0005fb0 comp rx=0 tx=0).stop 2024-01-24T13:35:27.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.931+0000 7f43cb66a700 1 -- 172.21.15.29:0/3009956260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43c4101210 msgr2=0x7f43c419dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:27.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.931+0000 7f43cb66a700 1 --2- 172.21.15.29:0/3009956260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43c4101210 0x7f43c419dfd0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f43bc00ef30 tx=0x7f43bc00c3a0 comp rx=0 tx=0).stop 2024-01-24T13:35:27.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.931+0000 7f43cb66a700 1 -- 172.21.15.29:0/3009956260 shutdown_connections 2024-01-24T13:35:27.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.931+0000 7f43cb66a700 1 --2- 172.21.15.29:0/3009956260 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f43b00a05f0 0x7f43b00a2ab0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:27.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.931+0000 7f43cb66a700 1 --2- 172.21.15.29:0/3009956260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43c4101210 0x7f43c419dfd0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:27.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.931+0000 7f43cb66a700 1 --2- 172.21.15.29:0/3009956260 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43c40ffeb0 0x7f43c4196a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:27.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.931+0000 7f43cb66a700 1 --2- 172.21.15.29:0/3009956260 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43c4100850 0x7f43c4196f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:27.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.931+0000 7f43cb66a700 1 -- 172.21.15.29:0/3009956260 >> 172.21.15.29:0/3009956260 conn(0x7f43c40fa900 msgr2=0x7f43c40fede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:27.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.931+0000 7f43cb66a700 1 -- 172.21.15.29:0/3009956260 shutdown_connections 2024-01-24T13:35:27.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:27.931+0000 7f43cb66a700 1 -- 172.21.15.29:0/3009956260 wait complete. 2024-01-24T13:35:27.944 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-24T13:35:27.945 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-24T13:35:28.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.088+0000 7fde2c664700 1 -- 172.21.15.29:0/1741860458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde24100d60 msgr2=0x7fde24101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:28.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.088+0000 7fde2c664700 1 --2- 172.21.15.29:0/1741860458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde24100d60 0x7fde24101180 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fde200099c0 tx=0x7fde20009cd0 comp rx=0 tx=0).stop 2024-01-24T13:35:28.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.089+0000 7fde2c664700 1 -- 172.21.15.29:0/1741860458 shutdown_connections 2024-01-24T13:35:28.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.089+0000 7fde2c664700 1 --2- 172.21.15.29:0/1741860458 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde241028f0 0x7fde241091a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:28.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.089+0000 7fde2c664700 1 --2- 172.21.15.29:0/1741860458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde24100d60 0x7fde24101180 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:28.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.089+0000 7fde2c664700 1 --2- 172.21.15.29:0/1741860458 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fde24101f50 0x7fde241023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:28.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.089+0000 7fde2c664700 1 -- 172.21.15.29:0/1741860458 >> 172.21.15.29:0/1741860458 conn(0x7fde240fc2f0 msgr2=0x7fde240fe710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:28.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.089+0000 7fde2c664700 1 -- 172.21.15.29:0/1741860458 shutdown_connections 2024-01-24T13:35:28.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.089+0000 7fde2c664700 1 -- 172.21.15.29:0/1741860458 wait complete. 2024-01-24T13:35:28.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.089+0000 7fde2c664700 1 Processor -- start 2024-01-24T13:35:28.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.089+0000 7fde2c664700 1 -- start start 2024-01-24T13:35:28.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.090+0000 7fde2c664700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde24100d60 0x7fde24196a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:28.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.090+0000 7fde2c664700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fde24101f50 0x7fde24196f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:28.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.090+0000 7fde2c664700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde241028f0 0x7fde2419dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:28.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.090+0000 7fde2c664700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde2419e5a0 con 0x7fde24100d60 2024-01-24T13:35:28.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.090+0000 7fde2c664700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde2419e6e0 con 0x7fde24101f50 2024-01-24T13:35:28.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.090+0000 7fde2c664700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde2419e870 con 0x7fde241028f0 2024-01-24T13:35:28.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.090+0000 7fde2a400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde24100d60 0x7fde24196a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:28.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.090+0000 7fde2a400700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde24100d60 0x7fde24196a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59210/0 (socket says 172.21.15.29:59210) 2024-01-24T13:35:28.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.090+0000 7fde2a400700 1 -- 172.21.15.29:0/3041882066 learned_addr learned my addr 172.21.15.29:0/3041882066 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:28.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.090+0000 7fde29bff700 1 --2- 172.21.15.29:0/3041882066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fde24101f50 0x7fde24196f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:28.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.090+0000 7fde2ac01700 1 --2- 172.21.15.29:0/3041882066 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde241028f0 0x7fde2419dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:28.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.090+0000 7fde2a400700 1 -- 172.21.15.29:0/3041882066 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde241028f0 msgr2=0x7fde2419dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:28.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.090+0000 7fde2a400700 1 --2- 172.21.15.29:0/3041882066 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde241028f0 0x7fde2419dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:28.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.090+0000 7fde2a400700 1 -- 172.21.15.29:0/3041882066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fde24101f50 msgr2=0x7fde24196f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:28.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.090+0000 7fde2a400700 1 --2- 172.21.15.29:0/3041882066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fde24101f50 0x7fde24196f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:28.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.090+0000 7fde2a400700 1 -- 172.21.15.29:0/3041882066 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde200095f0 con 0x7fde24100d60 2024-01-24T13:35:28.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.090+0000 7fde2ac01700 1 --2- 172.21.15.29:0/3041882066 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde241028f0 0x7fde2419dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T13:35:28.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.090+0000 7fde2a400700 1 --2- 172.21.15.29:0/3041882066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde24100d60 0x7fde24196a20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fde20009f50 tx=0x7fde20005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:28.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.090+0000 7fde1b7fe700 1 -- 172.21.15.29:0/3041882066 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde2001d070 con 0x7fde24100d60 2024-01-24T13:35:28.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.090+0000 7fde1b7fe700 1 -- 172.21.15.29:0/3041882066 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fde20004010 con 0x7fde24100d60 2024-01-24T13:35:28.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.090+0000 7fde2c664700 1 -- 172.21.15.29:0/3041882066 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde2419eaf0 con 0x7fde24100d60 2024-01-24T13:35:28.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.090+0000 7fde1b7fe700 1 -- 172.21.15.29:0/3041882066 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde2000fdc0 con 0x7fde24100d60 2024-01-24T13:35:28.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.091+0000 7fde2c664700 1 -- 172.21.15.29:0/3041882066 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde2419ef70 con 0x7fde24100d60 2024-01-24T13:35:28.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.092+0000 7fde2c664700 1 -- 172.21.15.29:0/3041882066 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde2410c5c0 con 0x7fde24100d60 2024-01-24T13:35:28.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.093+0000 7fde1b7fe700 1 -- 172.21.15.29:0/3041882066 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7fde2000f4f0 con 0x7fde24100d60 2024-01-24T13:35:28.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.093+0000 7fde1b7fe700 1 --2- 172.21.15.29:0/3041882066 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fde100a05a0 0x7fde100a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:28.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.094+0000 7fde1b7fe700 1 -- 172.21.15.29:0/3041882066 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fde200cc360 con 0x7fde24100d60 2024-01-24T13:35:28.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.094+0000 7fde29bff700 1 --2- 172.21.15.29:0/3041882066 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fde100a05a0 0x7fde100a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:28.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.094+0000 7fde29bff700 1 --2- 172.21.15.29:0/3041882066 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fde100a05a0 0x7fde100a2a60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fde14005950 tx=0x7fde1400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:28.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.098+0000 7fde1b7fe700 1 -- 172.21.15.29:0/3041882066 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fde20096430 con 0x7fde24100d60 2024-01-24T13:35:28.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:27 smithi163 ceph-mon[44969]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:28.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:27 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/1401677170' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T13:35:28.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:27 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/885656037' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-24T13:35:28.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:27 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:28.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:27 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:28.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:27 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:28.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:27 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:28.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:27 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:28.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:27 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:28.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:27 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:28.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:27 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:35:28.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:27 smithi163 ceph-mon[44969]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:28.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:27 smithi187 ceph-mon[44911]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:28.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:27 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/1401677170' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T13:35:28.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:27 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/885656037' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-24T13:35:28.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:27 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:28.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:27 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:28.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:27 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:28.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:27 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:28.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:27 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:28.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:27 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:28.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:27 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:28.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:27 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:35:28.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:27 smithi187 ceph-mon[44911]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:28.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.311+0000 7fde2c664700 1 -- 172.21.15.29:0/3041882066 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fde2410c780 con 0x7fde24100d60 2024-01-24T13:35:28.351 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:27 smithi029 ceph-mon[40651]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:28.351 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:27 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/1401677170' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T13:35:28.351 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:27 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/885656037' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-24T13:35:28.352 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:27 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:28.352 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:27 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:28.352 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:27 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:28.352 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:27 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:28.352 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:27 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:28.352 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:27 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:28.352 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:27 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T13:35:28.352 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:27 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T13:35:28.352 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:27 smithi029 ceph-mon[40651]: from='mgr.14152 172.21.15.29:0/4083614802' entity='mgr.x' 2024-01-24T13:35:28.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.922+0000 7fde1b7fe700 1 -- 172.21.15.29:0/3041882066 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v78) v1 ==== 138+0+0 (secure 0 0 0) 0x7fde20099a50 con 0x7fde24100d60 2024-01-24T13:35:28.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.926+0000 7fde2c664700 1 -- 172.21.15.29:0/3041882066 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fde100a05a0 msgr2=0x7fde100a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:28.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.926+0000 7fde2c664700 1 --2- 172.21.15.29:0/3041882066 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fde100a05a0 0x7fde100a2a60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fde14005950 tx=0x7fde1400b410 comp rx=0 tx=0).stop 2024-01-24T13:35:28.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.926+0000 7fde2c664700 1 -- 172.21.15.29:0/3041882066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde24100d60 msgr2=0x7fde24196a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:28.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.926+0000 7fde2c664700 1 --2- 172.21.15.29:0/3041882066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde24100d60 0x7fde24196a20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fde20009f50 tx=0x7fde20005740 comp rx=0 tx=0).stop 2024-01-24T13:35:28.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.927+0000 7fde2c664700 1 -- 172.21.15.29:0/3041882066 shutdown_connections 2024-01-24T13:35:28.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.927+0000 7fde2c664700 1 --2- 172.21.15.29:0/3041882066 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde241028f0 0x7fde2419dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:28.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.927+0000 7fde2c664700 1 --2- 172.21.15.29:0/3041882066 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fde100a05a0 0x7fde100a2a60 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:28.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.927+0000 7fde2c664700 1 --2- 172.21.15.29:0/3041882066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde24100d60 0x7fde24196a20 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:28.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.927+0000 7fde2c664700 1 --2- 172.21.15.29:0/3041882066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fde24101f50 0x7fde24196f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:28.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.927+0000 7fde2c664700 1 -- 172.21.15.29:0/3041882066 >> 172.21.15.29:0/3041882066 conn(0x7fde240fc2f0 msgr2=0x7fde240fe540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:28.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.927+0000 7fde2c664700 1 -- 172.21.15.29:0/3041882066 shutdown_connections 2024-01-24T13:35:28.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:28.927+0000 7fde2c664700 1 -- 172.21.15.29:0/3041882066 wait complete. 2024-01-24T13:35:28.930 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2024-01-24T13:35:28.945 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-01-24T13:35:29.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.092+0000 7f6f5aebf700 1 -- 172.21.15.29:0/402941755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f54100e20 msgr2=0x7f6f54101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:29.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.092+0000 7f6f5aebf700 1 --2- 172.21.15.29:0/402941755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f54100e20 0x7f6f54101280 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6f400099c0 tx=0x7f6f40009cd0 comp rx=0 tx=0).stop 2024-01-24T13:35:29.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.093+0000 7f6f5aebf700 1 -- 172.21.15.29:0/402941755 shutdown_connections 2024-01-24T13:35:29.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.093+0000 7f6f5aebf700 1 --2- 172.21.15.29:0/402941755 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f541017c0 0x7f6f54108070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:29.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.093+0000 7f6f5aebf700 1 --2- 172.21.15.29:0/402941755 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6f540ffc30 0x7f6f54100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:29.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.093+0000 7f6f5aebf700 1 --2- 172.21.15.29:0/402941755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f54100e20 0x7f6f54101280 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:29.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.093+0000 7f6f5aebf700 1 -- 172.21.15.29:0/402941755 >> 172.21.15.29:0/402941755 conn(0x7f6f540fb1a0 msgr2=0x7f6f540fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:29.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.093+0000 7f6f5aebf700 1 -- 172.21.15.29:0/402941755 shutdown_connections 2024-01-24T13:35:29.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.093+0000 7f6f5aebf700 1 -- 172.21.15.29:0/402941755 wait complete. 2024-01-24T13:35:29.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.093+0000 7f6f5aebf700 1 Processor -- start 2024-01-24T13:35:29.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.093+0000 7f6f5aebf700 1 -- start start 2024-01-24T13:35:29.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.093+0000 7f6f5aebf700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f540ffc30 0x7f6f54068a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:29.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.093+0000 7f6f5aebf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6f54100e20 0x7f6f5419b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:29.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.093+0000 7f6f5aebf700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f541017c0 0x7f6f5419ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:29.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.093+0000 7f6f5aebf700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5419e570 con 0x7f6f540ffc30 2024-01-24T13:35:29.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.093+0000 7f6f5aebf700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5419e6e0 con 0x7f6f541017c0 2024-01-24T13:35:29.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.093+0000 7f6f5aebf700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5419e8a0 con 0x7f6f54100e20 2024-01-24T13:35:29.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.094+0000 7f6f58c5b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f540ffc30 0x7f6f54068a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:29.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.094+0000 7f6f58c5b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f540ffc30 0x7f6f54068a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59226/0 (socket says 172.21.15.29:59226) 2024-01-24T13:35:29.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.094+0000 7f6f58c5b700 1 -- 172.21.15.29:0/2317081617 learned_addr learned my addr 172.21.15.29:0/2317081617 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:29.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.094+0000 7f6f53fff700 1 --2- 172.21.15.29:0/2317081617 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6f54100e20 0x7f6f5419b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:29.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.094+0000 7f6f5945c700 1 --2- 172.21.15.29:0/2317081617 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f541017c0 0x7f6f5419ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:29.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.094+0000 7f6f58c5b700 1 -- 172.21.15.29:0/2317081617 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6f54100e20 msgr2=0x7f6f5419b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:29.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.094+0000 7f6f58c5b700 1 --2- 172.21.15.29:0/2317081617 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6f54100e20 0x7f6f5419b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:29.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.094+0000 7f6f58c5b700 1 -- 172.21.15.29:0/2317081617 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f541017c0 msgr2=0x7f6f5419ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:29.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.094+0000 7f6f58c5b700 1 --2- 172.21.15.29:0/2317081617 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f541017c0 0x7f6f5419ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:29.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.094+0000 7f6f58c5b700 1 -- 172.21.15.29:0/2317081617 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f400095f0 con 0x7f6f540ffc30 2024-01-24T13:35:29.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.094+0000 7f6f5945c700 1 --2- 172.21.15.29:0/2317081617 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f541017c0 0x7f6f5419ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T13:35:29.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.094+0000 7f6f58c5b700 1 --2- 172.21.15.29:0/2317081617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f540ffc30 0x7f6f54068a10 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6f4800d980 tx=0x7f6f4800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:29.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.094+0000 7f6f51ffb700 1 -- 172.21.15.29:0/2317081617 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f48009930 con 0x7f6f540ffc30 2024-01-24T13:35:29.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.094+0000 7f6f5aebf700 1 -- 172.21.15.29:0/2317081617 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f5419eb80 con 0x7f6f540ffc30 2024-01-24T13:35:29.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.094+0000 7f6f51ffb700 1 -- 172.21.15.29:0/2317081617 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6f48010460 con 0x7f6f540ffc30 2024-01-24T13:35:29.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.094+0000 7f6f51ffb700 1 -- 172.21.15.29:0/2317081617 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4800f620 con 0x7f6f540ffc30 2024-01-24T13:35:29.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.095+0000 7f6f5aebf700 1 -- 172.21.15.29:0/2317081617 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f5419f040 con 0x7f6f540ffc30 2024-01-24T13:35:29.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.096+0000 7f6f5aebf700 1 -- 172.21.15.29:0/2317081617 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f54106bb0 con 0x7f6f540ffc30 2024-01-24T13:35:29.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.097+0000 7f6f51ffb700 1 -- 172.21.15.29:0/2317081617 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f6f48010860 con 0x7f6f540ffc30 2024-01-24T13:35:29.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.097+0000 7f6f51ffb700 1 --2- 172.21.15.29:0/2317081617 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f6f440a0520 0x7f6f440a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:29.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.098+0000 7f6f53fff700 1 --2- 172.21.15.29:0/2317081617 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f6f440a0520 0x7f6f440a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:29.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.098+0000 7f6f51ffb700 1 -- 172.21.15.29:0/2317081617 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f6f480cb1e0 con 0x7f6f540ffc30 2024-01-24T13:35:29.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.098+0000 7f6f53fff700 1 --2- 172.21.15.29:0/2317081617 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f6f440a0520 0x7f6f440a29e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6f4000b580 tx=0x7f6f40005bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:29.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.102+0000 7f6f51ffb700 1 -- 172.21.15.29:0/2317081617 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f6f48095010 con 0x7f6f540ffc30 2024-01-24T13:35:29.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:28 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/3009956260' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-01-24T13:35:29.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:28 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/3041882066' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-01-24T13:35:29.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:28 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/3009956260' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-01-24T13:35:29.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:28 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/3041882066' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-01-24T13:35:29.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.313+0000 7f6f5aebf700 1 -- 172.21.15.29:0/2317081617 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f6f5402cc70 con 0x7f6f540ffc30 2024-01-24T13:35:29.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:28 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/3009956260' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-01-24T13:35:29.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:28 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/3041882066' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-01-24T13:35:29.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.931+0000 7f6f51ffb700 1 -- 172.21.15.29:0/2317081617 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v79) v1 ==== 171+0+0 (secure 0 0 0) 0x7f6f48098630 con 0x7f6f540ffc30 2024-01-24T13:35:29.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.944+0000 7f6f5aebf700 1 -- 172.21.15.29:0/2317081617 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f6f440a0520 msgr2=0x7f6f440a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:29.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.944+0000 7f6f5aebf700 1 --2- 172.21.15.29:0/2317081617 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f6f440a0520 0x7f6f440a29e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6f4000b580 tx=0x7f6f40005bc0 comp rx=0 tx=0).stop 2024-01-24T13:35:29.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.944+0000 7f6f5aebf700 1 -- 172.21.15.29:0/2317081617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f540ffc30 msgr2=0x7f6f54068a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:29.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.944+0000 7f6f5aebf700 1 --2- 172.21.15.29:0/2317081617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f540ffc30 0x7f6f54068a10 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6f4800d980 tx=0x7f6f4800dc90 comp rx=0 tx=0).stop 2024-01-24T13:35:29.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.947+0000 7f6f5aebf700 1 -- 172.21.15.29:0/2317081617 shutdown_connections 2024-01-24T13:35:29.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.947+0000 7f6f5aebf700 1 --2- 172.21.15.29:0/2317081617 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f541017c0 0x7f6f5419ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:29.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.947+0000 7f6f5aebf700 1 --2- 172.21.15.29:0/2317081617 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f6f440a0520 0x7f6f440a29e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:29.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.947+0000 7f6f5aebf700 1 --2- 172.21.15.29:0/2317081617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f540ffc30 0x7f6f54068a10 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:29.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.947+0000 7f6f5aebf700 1 --2- 172.21.15.29:0/2317081617 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6f54100e20 0x7f6f5419b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:29.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.947+0000 7f6f5aebf700 1 -- 172.21.15.29:0/2317081617 >> 172.21.15.29:0/2317081617 conn(0x7f6f540fb1a0 msgr2=0x7f6f540fd480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:29.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.948+0000 7f6f5aebf700 1 -- 172.21.15.29:0/2317081617 shutdown_connections 2024-01-24T13:35:29.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:29.951+0000 7f6f5aebf700 1 -- 172.21.15.29:0/2317081617 wait complete. 2024-01-24T13:35:29.953 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2024-01-24T13:35:29.966 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-24T13:35:30.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.107+0000 7f74f492d700 1 -- 172.21.15.29:0/2465076548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74f0106a70 msgr2=0x7f74f0108e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:30.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.107+0000 7f74f492d700 1 --2- 172.21.15.29:0/2465076548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74f0106a70 0x7f74f0108e80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f74e800b0a0 tx=0x7f74e800b3b0 comp rx=0 tx=0).stop 2024-01-24T13:35:30.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.107+0000 7f74f492d700 1 -- 172.21.15.29:0/2465076548 shutdown_connections 2024-01-24T13:35:30.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.107+0000 7f74f492d700 1 --2- 172.21.15.29:0/2465076548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74f0106a70 0x7f74f0108e80 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:30.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.107+0000 7f74f492d700 1 --2- 172.21.15.29:0/2465076548 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74f0101810 0x7f74f0103c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:30.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.107+0000 7f74f492d700 1 --2- 172.21.15.29:0/2465076548 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f74f0104140 0x7f74f0106530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:30.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.107+0000 7f74f492d700 1 -- 172.21.15.29:0/2465076548 >> 172.21.15.29:0/2465076548 conn(0x7f74f00fb1a0 msgr2=0x7f74f00fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:30.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.107+0000 7f74f492d700 1 -- 172.21.15.29:0/2465076548 shutdown_connections 2024-01-24T13:35:30.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.107+0000 7f74f492d700 1 -- 172.21.15.29:0/2465076548 wait complete. 2024-01-24T13:35:30.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.108+0000 7f74f492d700 1 Processor -- start 2024-01-24T13:35:30.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.108+0000 7f74f492d700 1 -- start start 2024-01-24T13:35:30.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.108+0000 7f74f492d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74f0101810 0x7f74f0198d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:30.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.109+0000 7f74f492d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74f0104140 0x7f74f0199260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:30.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.109+0000 7f74f492d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f74f0106a70 0x7f74f01a0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:30.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.109+0000 7f74f492d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f01a0860 con 0x7f74f0101810 2024-01-24T13:35:30.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.109+0000 7f74f492d700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f01a09d0 con 0x7f74f0106a70 2024-01-24T13:35:30.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.109+0000 7f74f492d700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f01a0b90 con 0x7f74f0104140 2024-01-24T13:35:30.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.109+0000 7f74ee59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74f0101810 0x7f74f0198d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:30.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.109+0000 7f74ee59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74f0101810 0x7f74f0198d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59238/0 (socket says 172.21.15.29:59238) 2024-01-24T13:35:30.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.109+0000 7f74ee59c700 1 -- 172.21.15.29:0/4241928202 learned_addr learned my addr 172.21.15.29:0/4241928202 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:30.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.109+0000 7f74eed9d700 1 --2- 172.21.15.29:0/4241928202 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f74f0106a70 0x7f74f01a0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:30.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.109+0000 7f74e5d9b700 1 --2- 172.21.15.29:0/4241928202 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74f0104140 0x7f74f0199260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:30.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.109+0000 7f74ee59c700 1 -- 172.21.15.29:0/4241928202 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74f0104140 msgr2=0x7f74f0199260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:30.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.109+0000 7f74ee59c700 1 --2- 172.21.15.29:0/4241928202 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74f0104140 0x7f74f0199260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:30.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.109+0000 7f74ee59c700 1 -- 172.21.15.29:0/4241928202 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f74f0106a70 msgr2=0x7f74f01a0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:30.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.109+0000 7f74ee59c700 1 --2- 172.21.15.29:0/4241928202 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f74f0106a70 0x7f74f01a0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:30.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.109+0000 7f74ee59c700 1 -- 172.21.15.29:0/4241928202 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74e8009d00 con 0x7f74f0101810 2024-01-24T13:35:30.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.109+0000 7f74eed9d700 1 --2- 172.21.15.29:0/4241928202 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f74f0106a70 0x7f74f01a0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T13:35:30.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.109+0000 7f74e5d9b700 1 --2- 172.21.15.29:0/4241928202 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74f0104140 0x7f74f0199260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T13:35:30.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.109+0000 7f74ee59c700 1 --2- 172.21.15.29:0/4241928202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74f0101810 0x7f74f0198d20 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f74e000d980 tx=0x7f74e000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:30.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.109+0000 7f74e7fff700 1 -- 172.21.15.29:0/4241928202 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e0004220 con 0x7f74f0101810 2024-01-24T13:35:30.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.109+0000 7f74e7fff700 1 -- 172.21.15.29:0/4241928202 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f74e0007640 con 0x7f74f0101810 2024-01-24T13:35:30.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.109+0000 7f74f492d700 1 -- 172.21.15.29:0/4241928202 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74f01a0e70 con 0x7f74f0101810 2024-01-24T13:35:30.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.109+0000 7f74e7fff700 1 -- 172.21.15.29:0/4241928202 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e0005610 con 0x7f74f0101810 2024-01-24T13:35:30.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.110+0000 7f74f492d700 1 -- 172.21.15.29:0/4241928202 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74f01a1290 con 0x7f74f0101810 2024-01-24T13:35:30.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.111+0000 7f74f492d700 1 -- 172.21.15.29:0/4241928202 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74f01085f0 con 0x7f74f0101810 2024-01-24T13:35:30.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.112+0000 7f74e7fff700 1 -- 172.21.15.29:0/4241928202 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f74e0004750 con 0x7f74f0101810 2024-01-24T13:35:30.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.113+0000 7f74e7fff700 1 --2- 172.21.15.29:0/4241928202 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f74dc0a0520 0x7f74dc0a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:30.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.113+0000 7f74e5d9b700 1 --2- 172.21.15.29:0/4241928202 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f74dc0a0520 0x7f74dc0a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:30.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.113+0000 7f74e7fff700 1 -- 172.21.15.29:0/4241928202 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7879+0+0 (secure 0 0 0) 0x7f74e00cbbb0 con 0x7f74f0101810 2024-01-24T13:35:30.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.114+0000 7f74e5d9b700 1 --2- 172.21.15.29:0/4241928202 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f74dc0a0520 0x7f74dc0a29e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f74d8005e10 tx=0x7f74d8005d60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:30.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.117+0000 7f74e7fff700 1 -- 172.21.15.29:0/4241928202 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f74e0095860 con 0x7f74f0101810 2024-01-24T13:35:30.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:29 smithi163 ceph-mon[44969]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:30.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:29 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/3041882066' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-01-24T13:35:30.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:29 smithi163 ceph-mon[44969]: osdmap e78: 12 total, 12 up, 12 in 2024-01-24T13:35:30.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:29 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/2317081617' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-01-24T13:35:30.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:29 smithi187 ceph-mon[44911]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:30.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:29 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/3041882066' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-01-24T13:35:30.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:29 smithi187 ceph-mon[44911]: osdmap e78: 12 total, 12 up, 12 in 2024-01-24T13:35:30.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:29 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/2317081617' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-01-24T13:35:30.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:29 smithi029 ceph-mon[40651]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:30.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:29 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/3041882066' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-01-24T13:35:30.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:29 smithi029 ceph-mon[40651]: osdmap e78: 12 total, 12 up, 12 in 2024-01-24T13:35:30.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:29 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/2317081617' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-01-24T13:35:30.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.389+0000 7f74f492d700 1 -- 172.21.15.29:0/4241928202 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f74f002cc70 con 0x7f74f0101810 2024-01-24T13:35:30.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.396+0000 7f74e7fff700 1 -- 172.21.15.29:0/4241928202 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f74e0098e80 con 0x7f74f0101810 2024-01-24T13:35:30.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.399+0000 7f74f492d700 1 -- 172.21.15.29:0/4241928202 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f74dc0a0520 msgr2=0x7f74dc0a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:30.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.399+0000 7f74f492d700 1 --2- 172.21.15.29:0/4241928202 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f74dc0a0520 0x7f74dc0a29e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f74d8005e10 tx=0x7f74d8005d60 comp rx=0 tx=0).stop 2024-01-24T13:35:30.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.399+0000 7f74f492d700 1 -- 172.21.15.29:0/4241928202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74f0101810 msgr2=0x7f74f0198d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:30.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.399+0000 7f74f492d700 1 --2- 172.21.15.29:0/4241928202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74f0101810 0x7f74f0198d20 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f74e000d980 tx=0x7f74e000dc90 comp rx=0 tx=0).stop 2024-01-24T13:35:30.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.400+0000 7f74f492d700 1 -- 172.21.15.29:0/4241928202 shutdown_connections 2024-01-24T13:35:30.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.400+0000 7f74f492d700 1 --2- 172.21.15.29:0/4241928202 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f74f0106a70 0x7f74f01a0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:30.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.400+0000 7f74f492d700 1 --2- 172.21.15.29:0/4241928202 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f74dc0a0520 0x7f74dc0a29e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:30.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.400+0000 7f74f492d700 1 --2- 172.21.15.29:0/4241928202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74f0101810 0x7f74f0198d20 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:30.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.400+0000 7f74f492d700 1 --2- 172.21.15.29:0/4241928202 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74f0104140 0x7f74f0199260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:30.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.400+0000 7f74f492d700 1 -- 172.21.15.29:0/4241928202 >> 172.21.15.29:0/4241928202 conn(0x7f74f00fb1a0 msgr2=0x7f74f00ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:30.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.400+0000 7f74f492d700 1 -- 172.21.15.29:0/4241928202 shutdown_connections 2024-01-24T13:35:30.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.402+0000 7f74f492d700 1 -- 172.21.15.29:0/4241928202 wait complete. 2024-01-24T13:35:30.420 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 2 and data pool 3 2024-01-24T13:35:30.437 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-01-24T13:35:30.437 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-01-24T13:35:30.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.580+0000 7fb6b06b9700 1 -- 172.21.15.29:0/1195086304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6a81017c0 msgr2=0x7fb6a8108070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:30.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.580+0000 7fb6b06b9700 1 --2- 172.21.15.29:0/1195086304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6a81017c0 0x7fb6a8108070 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb6a000b0d0 tx=0x7fb6a000b3e0 comp rx=0 tx=0).stop 2024-01-24T13:35:30.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.580+0000 7fb6b06b9700 1 -- 172.21.15.29:0/1195086304 shutdown_connections 2024-01-24T13:35:30.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.580+0000 7fb6b06b9700 1 --2- 172.21.15.29:0/1195086304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6a81017c0 0x7fb6a8108070 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:30.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.580+0000 7fb6b06b9700 1 --2- 172.21.15.29:0/1195086304 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6a80ffc30 0x7fb6a8100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:30.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.580+0000 7fb6b06b9700 1 --2- 172.21.15.29:0/1195086304 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb6a8100e20 0x7fb6a8101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:30.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.580+0000 7fb6b06b9700 1 -- 172.21.15.29:0/1195086304 >> 172.21.15.29:0/1195086304 conn(0x7fb6a80fb1a0 msgr2=0x7fb6a80fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:30.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.580+0000 7fb6b06b9700 1 -- 172.21.15.29:0/1195086304 shutdown_connections 2024-01-24T13:35:30.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.580+0000 7fb6b06b9700 1 -- 172.21.15.29:0/1195086304 wait complete. 2024-01-24T13:35:30.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.580+0000 7fb6b06b9700 1 Processor -- start 2024-01-24T13:35:30.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.581+0000 7fb6b06b9700 1 -- start start 2024-01-24T13:35:30.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.581+0000 7fb6b06b9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb6a80ffc30 0x7fb6a8103bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:30.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.581+0000 7fb6b06b9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6a8100e20 0x7fb6a8102200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:30.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.581+0000 7fb6b06b9700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6a81017c0 0x7fb6a8102740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:30.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.581+0000 7fb6b06b9700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a8102ee0 con 0x7fb6a8100e20 2024-01-24T13:35:30.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.581+0000 7fb6b06b9700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a8103050 con 0x7fb6a81017c0 2024-01-24T13:35:30.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.581+0000 7fb6b06b9700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a81a0bd0 con 0x7fb6a80ffc30 2024-01-24T13:35:30.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.581+0000 7fb6adc54700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6a8100e20 0x7fb6a8102200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:30.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.581+0000 7fb6adc54700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6a8100e20 0x7fb6a8102200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59254/0 (socket says 172.21.15.29:59254) 2024-01-24T13:35:30.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.581+0000 7fb6adc54700 1 -- 172.21.15.29:0/694769948 learned_addr learned my addr 172.21.15.29:0/694769948 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:30.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.581+0000 7fb6aec56700 1 --2- 172.21.15.29:0/694769948 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6a81017c0 0x7fb6a8102740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:30.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.581+0000 7fb6ae455700 1 --2- 172.21.15.29:0/694769948 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb6a80ffc30 0x7fb6a8103bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:30.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.581+0000 7fb6adc54700 1 -- 172.21.15.29:0/694769948 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb6a80ffc30 msgr2=0x7fb6a8103bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:30.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.581+0000 7fb6adc54700 1 --2- 172.21.15.29:0/694769948 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb6a80ffc30 0x7fb6a8103bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:30.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.581+0000 7fb6adc54700 1 -- 172.21.15.29:0/694769948 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6a81017c0 msgr2=0x7fb6a8102740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:30.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.581+0000 7fb6adc54700 1 --2- 172.21.15.29:0/694769948 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6a81017c0 0x7fb6a8102740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:30.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.581+0000 7fb6adc54700 1 -- 172.21.15.29:0/694769948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a0009d00 con 0x7fb6a8100e20 2024-01-24T13:35:30.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.581+0000 7fb6ae455700 1 --2- 172.21.15.29:0/694769948 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb6a80ffc30 0x7fb6a8103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T13:35:30.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.581+0000 7fb6adc54700 1 --2- 172.21.15.29:0/694769948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6a8100e20 0x7fb6a8102200 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb69800d950 tx=0x7fb69800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:30.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.582+0000 7fb69f7fe700 1 -- 172.21.15.29:0/694769948 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb698009940 con 0x7fb6a8100e20 2024-01-24T13:35:30.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.582+0000 7fb69f7fe700 1 -- 172.21.15.29:0/694769948 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb698010460 con 0x7fb6a8100e20 2024-01-24T13:35:30.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.582+0000 7fb6b06b9700 1 -- 172.21.15.29:0/694769948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6a81a0eb0 con 0x7fb6a8100e20 2024-01-24T13:35:30.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.582+0000 7fb69f7fe700 1 -- 172.21.15.29:0/694769948 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb69800f620 con 0x7fb6a8100e20 2024-01-24T13:35:30.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.582+0000 7fb6b06b9700 1 -- 172.21.15.29:0/694769948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6a81a1330 con 0x7fb6a8100e20 2024-01-24T13:35:30.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.583+0000 7fb6b06b9700 1 -- 172.21.15.29:0/694769948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6a810b4a0 con 0x7fb6a8100e20 2024-01-24T13:35:30.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.585+0000 7fb69f7fe700 1 -- 172.21.15.29:0/694769948 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7fb6980108d0 con 0x7fb6a8100e20 2024-01-24T13:35:30.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.585+0000 7fb69f7fe700 1 --2- 172.21.15.29:0/694769948 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb6940a0520 0x7fb6940a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:30.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.585+0000 7fb69f7fe700 1 -- 172.21.15.29:0/694769948 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fb6980cb660 con 0x7fb6a8100e20 2024-01-24T13:35:30.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.585+0000 7fb6ae455700 1 --2- 172.21.15.29:0/694769948 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb6940a0520 0x7fb6940a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:30.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.586+0000 7fb6ae455700 1 --2- 172.21.15.29:0/694769948 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb6940a0520 0x7fb6940a29e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb6a8100c80 tx=0x7fb6a4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:30.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.589+0000 7fb69f7fe700 1 -- 172.21.15.29:0/694769948 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fb698095350 con 0x7fb6a8100e20 2024-01-24T13:35:30.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:30.816+0000 7fb6b06b9700 1 -- 172.21.15.29:0/694769948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7fb6a81a1970 con 0x7fb6a8100e20 2024-01-24T13:35:31.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:30 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/2317081617' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-01-24T13:35:31.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:30 smithi163 ceph-mon[44969]: osdmap e79: 12 total, 12 up, 12 in 2024-01-24T13:35:31.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:30 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/4241928202' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-01-24T13:35:31.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:30 smithi163 ceph-mon[44969]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-01-24T13:35:31.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:30 smithi163 ceph-mon[44969]: osdmap e80: 12 total, 12 up, 12 in 2024-01-24T13:35:31.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:30 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/4241928202' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-01-24T13:35:31.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:30 smithi163 ceph-mon[44969]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-01-24T13:35:31.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:30 smithi163 ceph-mon[44969]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-01-24T13:35:31.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:30 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/694769948' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-01-24T13:35:31.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:30 smithi163 ceph-mon[44969]: mds.? [v2:172.21.15.163:6838/2641650917,v1:172.21.15.163:6839/2641650917] up:active 2024-01-24T13:35:31.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:30 smithi163 ceph-mon[44969]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-01-24T13:35:31.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:30 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/2317081617' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-01-24T13:35:31.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:30 smithi187 ceph-mon[44911]: osdmap e79: 12 total, 12 up, 12 in 2024-01-24T13:35:31.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:30 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/4241928202' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-01-24T13:35:31.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:30 smithi187 ceph-mon[44911]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-01-24T13:35:31.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:30 smithi187 ceph-mon[44911]: osdmap e80: 12 total, 12 up, 12 in 2024-01-24T13:35:31.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:30 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/4241928202' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-01-24T13:35:31.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:30 smithi187 ceph-mon[44911]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-01-24T13:35:31.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:30 smithi187 ceph-mon[44911]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-01-24T13:35:31.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:30 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/694769948' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-01-24T13:35:31.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:30 smithi187 ceph-mon[44911]: mds.? [v2:172.21.15.163:6838/2641650917,v1:172.21.15.163:6839/2641650917] up:active 2024-01-24T13:35:31.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:30 smithi187 ceph-mon[44911]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-01-24T13:35:31.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:30 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/2317081617' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-01-24T13:35:31.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:30 smithi029 ceph-mon[40651]: osdmap e79: 12 total, 12 up, 12 in 2024-01-24T13:35:31.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:30 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/4241928202' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-01-24T13:35:31.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:30 smithi029 ceph-mon[40651]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-01-24T13:35:31.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:30 smithi029 ceph-mon[40651]: osdmap e80: 12 total, 12 up, 12 in 2024-01-24T13:35:31.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:30 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/4241928202' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-01-24T13:35:31.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:30 smithi029 ceph-mon[40651]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-01-24T13:35:31.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:30 smithi029 ceph-mon[40651]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-01-24T13:35:31.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:30 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/694769948' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-01-24T13:35:31.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:30 smithi029 ceph-mon[40651]: mds.? [v2:172.21.15.163:6838/2641650917,v1:172.21.15.163:6839/2641650917] up:active 2024-01-24T13:35:31.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:30 smithi029 ceph-mon[40651]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-01-24T13:35:31.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.400+0000 7fb69f7fe700 1 -- 172.21.15.29:0/694769948 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v81) v1 ==== 157+0+0 (secure 0 0 0) 0x7fb6980160e0 con 0x7fb6a8100e20 2024-01-24T13:35:31.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.403+0000 7fb6b06b9700 1 -- 172.21.15.29:0/694769948 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb6940a0520 msgr2=0x7fb6940a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:31.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.403+0000 7fb6b06b9700 1 --2- 172.21.15.29:0/694769948 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb6940a0520 0x7fb6940a29e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb6a8100c80 tx=0x7fb6a4009260 comp rx=0 tx=0).stop 2024-01-24T13:35:31.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.404+0000 7fb6b06b9700 1 -- 172.21.15.29:0/694769948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6a8100e20 msgr2=0x7fb6a8102200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:31.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.404+0000 7fb6b06b9700 1 --2- 172.21.15.29:0/694769948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6a8100e20 0x7fb6a8102200 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb69800d950 tx=0x7fb69800dc60 comp rx=0 tx=0).stop 2024-01-24T13:35:31.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.404+0000 7fb6b06b9700 1 -- 172.21.15.29:0/694769948 shutdown_connections 2024-01-24T13:35:31.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.404+0000 7fb6b06b9700 1 --2- 172.21.15.29:0/694769948 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6a81017c0 0x7fb6a8102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:31.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.404+0000 7fb6b06b9700 1 --2- 172.21.15.29:0/694769948 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb6a80ffc30 0x7fb6a8103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:31.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.404+0000 7fb6b06b9700 1 --2- 172.21.15.29:0/694769948 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb6940a0520 0x7fb6940a29e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:31.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.404+0000 7fb6b06b9700 1 --2- 172.21.15.29:0/694769948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6a8100e20 0x7fb6a8102200 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:31.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.404+0000 7fb6b06b9700 1 -- 172.21.15.29:0/694769948 >> 172.21.15.29:0/694769948 conn(0x7fb6a80fb1a0 msgr2=0x7fb6a8067f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:31.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.404+0000 7fb6b06b9700 1 -- 172.21.15.29:0/694769948 shutdown_connections 2024-01-24T13:35:31.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.404+0000 7fb6b06b9700 1 -- 172.21.15.29:0/694769948 wait complete. 2024-01-24T13:35:31.420 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-01-24T13:35:31.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.570+0000 7f832f348700 1 -- 172.21.15.29:0/1607882014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83280ff010 msgr2=0x7f83280ff430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:31.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.570+0000 7f832f348700 1 --2- 172.21.15.29:0/1607882014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83280ff010 0x7f83280ff430 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8324009990 tx=0x7f8324009ca0 comp rx=0 tx=0).stop 2024-01-24T13:35:31.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.571+0000 7f832f348700 1 -- 172.21.15.29:0/1607882014 shutdown_connections 2024-01-24T13:35:31.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.571+0000 7f832f348700 1 --2- 172.21.15.29:0/1607882014 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8328102980 0x7f8328109230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:31.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.571+0000 7f832f348700 1 --2- 172.21.15.29:0/1607882014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83280ff010 0x7f83280ff430 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:31.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.571+0000 7f832f348700 1 --2- 172.21.15.29:0/1607882014 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8328101fe0 0x7f8328102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:31.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.571+0000 7f832f348700 1 -- 172.21.15.29:0/1607882014 >> 172.21.15.29:0/1607882014 conn(0x7f83280fa920 msgr2=0x7f83280fcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:31.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.571+0000 7f832f348700 1 -- 172.21.15.29:0/1607882014 shutdown_connections 2024-01-24T13:35:31.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.571+0000 7f832f348700 1 -- 172.21.15.29:0/1607882014 wait complete. 2024-01-24T13:35:31.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.571+0000 7f832f348700 1 Processor -- start 2024-01-24T13:35:31.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.571+0000 7f832f348700 1 -- start start 2024-01-24T13:35:31.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.571+0000 7f832f348700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83280ff010 0x7f8328196900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:31.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.571+0000 7f832f348700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8328101fe0 0x7f8328196e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:31.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.571+0000 7f832f348700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8328102980 0x7f832819deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:31.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.571+0000 7f832f348700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832819e480 con 0x7f8328102980 2024-01-24T13:35:31.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.571+0000 7f832f348700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832819e5c0 con 0x7f8328101fe0 2024-01-24T13:35:31.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.571+0000 7f832f348700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832819e750 con 0x7f83280ff010 2024-01-24T13:35:31.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.572+0000 7f832d8e5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8328102980 0x7f832819deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:31.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.572+0000 7f832d8e5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8328102980 0x7f832819deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52256/0 (socket says 172.21.15.29:52256) 2024-01-24T13:35:31.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.572+0000 7f832d8e5700 1 -- 172.21.15.29:0/1945596773 learned_addr learned my addr 172.21.15.29:0/1945596773 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:31.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.572+0000 7f832d0e4700 1 --2- 172.21.15.29:0/1945596773 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83280ff010 0x7f8328196900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:31.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.572+0000 7f832c8e3700 1 --2- 172.21.15.29:0/1945596773 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8328101fe0 0x7f8328196e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:31.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.572+0000 7f832d8e5700 1 -- 172.21.15.29:0/1945596773 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83280ff010 msgr2=0x7f8328196900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:31.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.572+0000 7f832d8e5700 1 --2- 172.21.15.29:0/1945596773 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83280ff010 0x7f8328196900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:31.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.572+0000 7f832d8e5700 1 -- 172.21.15.29:0/1945596773 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8328101fe0 msgr2=0x7f8328196e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:31.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.572+0000 7f832d8e5700 1 --2- 172.21.15.29:0/1945596773 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8328101fe0 0x7f8328196e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:31.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.572+0000 7f832d8e5700 1 -- 172.21.15.29:0/1945596773 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83240095f0 con 0x7f8328102980 2024-01-24T13:35:31.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.572+0000 7f832d0e4700 1 --2- 172.21.15.29:0/1945596773 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83280ff010 0x7f8328196900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T13:35:31.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.572+0000 7f832d8e5700 1 --2- 172.21.15.29:0/1945596773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8328102980 0x7f832819deb0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8320007d70 tx=0x7f832000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:31.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.572+0000 7f831e7fc700 1 -- 172.21.15.29:0/1945596773 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832000ad40 con 0x7f8328102980 2024-01-24T13:35:31.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.572+0000 7f831e7fc700 1 -- 172.21.15.29:0/1945596773 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f83200128e0 con 0x7f8328102980 2024-01-24T13:35:31.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.572+0000 7f831e7fc700 1 -- 172.21.15.29:0/1945596773 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8320011780 con 0x7f8328102980 2024-01-24T13:35:31.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.572+0000 7f832f348700 1 -- 172.21.15.29:0/1945596773 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f832819ea30 con 0x7f8328102980 2024-01-24T13:35:31.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.572+0000 7f832f348700 1 -- 172.21.15.29:0/1945596773 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f832819eef0 con 0x7f8328102980 2024-01-24T13:35:31.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.573+0000 7f832f348700 1 -- 172.21.15.29:0/1945596773 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83281082c0 con 0x7f8328102980 2024-01-24T13:35:31.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.574+0000 7f831e7fc700 1 -- 172.21.15.29:0/1945596773 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f8320012a50 con 0x7f8328102980 2024-01-24T13:35:31.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.575+0000 7f831e7fc700 1 --2- 172.21.15.29:0/1945596773 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f83140a0640 0x7f83140a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:31.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.575+0000 7f832d0e4700 1 --2- 172.21.15.29:0/1945596773 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f83140a0640 0x7f83140a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:31.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.575+0000 7f831e7fc700 1 -- 172.21.15.29:0/1945596773 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8205+0+0 (secure 0 0 0) 0x7f83200cbc60 con 0x7f8328102980 2024-01-24T13:35:31.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.576+0000 7f832d0e4700 1 --2- 172.21.15.29:0/1945596773 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f83140a0640 0x7f83140a2b00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8324006010 tx=0x7f832400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:31.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.580+0000 7f831e7fc700 1 -- 172.21.15.29:0/1945596773 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f8320095820 con 0x7f8328102980 2024-01-24T13:35:31.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:31.788+0000 7f832f348700 1 -- 172.21.15.29:0/1945596773 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f832802cc70 con 0x7f8328102980 2024-01-24T13:35:32.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:31 smithi163 ceph-mon[44969]: pgmap v220: 97 pgs: 16 creating+peering, 4 active+clean, 77 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:32.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:31 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/694769948' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-01-24T13:35:32.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:31 smithi163 ceph-mon[44969]: osdmap e81: 12 total, 12 up, 12 in 2024-01-24T13:35:32.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:31 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/1945596773' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-01-24T13:35:32.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:31 smithi187 ceph-mon[44911]: pgmap v220: 97 pgs: 16 creating+peering, 4 active+clean, 77 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:32.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:31 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/694769948' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-01-24T13:35:32.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:31 smithi187 ceph-mon[44911]: osdmap e81: 12 total, 12 up, 12 in 2024-01-24T13:35:32.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:31 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/1945596773' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-01-24T13:35:32.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:31 smithi029 ceph-mon[40651]: pgmap v220: 97 pgs: 16 creating+peering, 4 active+clean, 77 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T13:35:32.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:31 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/694769948' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-01-24T13:35:32.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:31 smithi029 ceph-mon[40651]: osdmap e81: 12 total, 12 up, 12 in 2024-01-24T13:35:32.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:31 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/1945596773' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-01-24T13:35:33.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.412+0000 7f831e7fc700 1 -- 172.21.15.29:0/1945596773 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v83) v1 ==== 228+0+0 (secure 0 0 0) 0x7f8320098e40 con 0x7f8328102980 2024-01-24T13:35:33.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.418+0000 7f832f348700 1 -- 172.21.15.29:0/1945596773 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f83140a0640 msgr2=0x7f83140a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:33.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.418+0000 7f832f348700 1 --2- 172.21.15.29:0/1945596773 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f83140a0640 0x7f83140a2b00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8324006010 tx=0x7f832400b540 comp rx=0 tx=0).stop 2024-01-24T13:35:33.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.418+0000 7f832f348700 1 -- 172.21.15.29:0/1945596773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8328102980 msgr2=0x7f832819deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:33.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.418+0000 7f832f348700 1 --2- 172.21.15.29:0/1945596773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8328102980 0x7f832819deb0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8320007d70 tx=0x7f832000a340 comp rx=0 tx=0).stop 2024-01-24T13:35:33.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.418+0000 7f832f348700 1 -- 172.21.15.29:0/1945596773 shutdown_connections 2024-01-24T13:35:33.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.418+0000 7f832f348700 1 --2- 172.21.15.29:0/1945596773 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f83140a0640 0x7f83140a2b00 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:33.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.418+0000 7f832f348700 1 --2- 172.21.15.29:0/1945596773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8328102980 0x7f832819deb0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:33.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.418+0000 7f832f348700 1 --2- 172.21.15.29:0/1945596773 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83280ff010 0x7f8328196900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:33.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.418+0000 7f832f348700 1 --2- 172.21.15.29:0/1945596773 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8328101fe0 0x7f8328196e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:33.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.419+0000 7f832f348700 1 -- 172.21.15.29:0/1945596773 >> 172.21.15.29:0/1945596773 conn(0x7f83280fa920 msgr2=0x7f8328105400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:33.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.419+0000 7f832f348700 1 -- 172.21.15.29:0/1945596773 shutdown_connections 2024-01-24T13:35:33.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.419+0000 7f832f348700 1 -- 172.21.15.29:0/1945596773 wait complete. 2024-01-24T13:35:33.421 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data_ec' created 2024-01-24T13:35:33.435 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-01-24T13:35:33.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.580+0000 7fd2df741700 1 -- 172.21.15.29:0/2931112159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2d8100800 msgr2=0x7fd2d81080a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:33.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.580+0000 7fd2df741700 1 --2- 172.21.15.29:0/2931112159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2d8100800 0x7fd2d81080a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd2d000b0a0 tx=0x7fd2d000b3b0 comp rx=0 tx=0).stop 2024-01-24T13:35:33.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.580+0000 7fd2df741700 1 -- 172.21.15.29:0/2931112159 shutdown_connections 2024-01-24T13:35:33.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.580+0000 7fd2df741700 1 --2- 172.21.15.29:0/2931112159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2d8100800 0x7fd2d81080a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:33.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.580+0000 7fd2df741700 1 --2- 172.21.15.29:0/2931112159 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2d81011f0 0x7fd2d8101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:33.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.580+0000 7fd2df741700 1 --2- 172.21.15.29:0/2931112159 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd2d80ffe40 0x7fd2d81002c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:33.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.580+0000 7fd2df741700 1 -- 172.21.15.29:0/2931112159 >> 172.21.15.29:0/2931112159 conn(0x7fd2d80fa910 msgr2=0x7fd2d80fcd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:33.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.580+0000 7fd2df741700 1 -- 172.21.15.29:0/2931112159 shutdown_connections 2024-01-24T13:35:33.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.580+0000 7fd2df741700 1 -- 172.21.15.29:0/2931112159 wait complete. 2024-01-24T13:35:33.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.580+0000 7fd2df741700 1 Processor -- start 2024-01-24T13:35:33.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.580+0000 7fd2df741700 1 -- start start 2024-01-24T13:35:33.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.581+0000 7fd2df741700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2d80ffe40 0x7fd2d8103bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:33.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.581+0000 7fd2df741700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd2d8100800 0x7fd2d8102230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:33.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.581+0000 7fd2df741700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2d81011f0 0x7fd2d8102770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:33.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.581+0000 7fd2df741700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d8102f10 con 0x7fd2d80ffe40 2024-01-24T13:35:33.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.581+0000 7fd2df741700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d8103080 con 0x7fd2d8100800 2024-01-24T13:35:33.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.581+0000 7fd2df741700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d81a0b90 con 0x7fd2d81011f0 2024-01-24T13:35:33.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.581+0000 7fd2dd4dd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2d80ffe40 0x7fd2d8103bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:33.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.581+0000 7fd2dd4dd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2d80ffe40 0x7fd2d8103bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52278/0 (socket says 172.21.15.29:52278) 2024-01-24T13:35:33.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.581+0000 7fd2dd4dd700 1 -- 172.21.15.29:0/2508210489 learned_addr learned my addr 172.21.15.29:0/2508210489 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:33.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.581+0000 7fd2dccdc700 1 --2- 172.21.15.29:0/2508210489 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd2d8100800 0x7fd2d8102230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:33.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.581+0000 7fd2ddcde700 1 --2- 172.21.15.29:0/2508210489 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2d81011f0 0x7fd2d8102770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:33.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.581+0000 7fd2dd4dd700 1 -- 172.21.15.29:0/2508210489 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2d81011f0 msgr2=0x7fd2d8102770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:33.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.581+0000 7fd2dd4dd700 1 --2- 172.21.15.29:0/2508210489 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2d81011f0 0x7fd2d8102770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:33.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.581+0000 7fd2dd4dd700 1 -- 172.21.15.29:0/2508210489 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd2d8100800 msgr2=0x7fd2d8102230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:33.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.581+0000 7fd2dd4dd700 1 --2- 172.21.15.29:0/2508210489 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd2d8100800 0x7fd2d8102230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:33.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.581+0000 7fd2dd4dd700 1 -- 172.21.15.29:0/2508210489 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d0009d00 con 0x7fd2d80ffe40 2024-01-24T13:35:33.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.581+0000 7fd2dd4dd700 1 --2- 172.21.15.29:0/2508210489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2d80ffe40 0x7fd2d8103bf0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd2d400d980 tx=0x7fd2d400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:33.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.582+0000 7fd2ce7fc700 1 -- 172.21.15.29:0/2508210489 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d4004020 con 0x7fd2d80ffe40 2024-01-24T13:35:33.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.582+0000 7fd2df741700 1 -- 172.21.15.29:0/2508210489 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2d81a0e70 con 0x7fd2d80ffe40 2024-01-24T13:35:33.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.582+0000 7fd2ce7fc700 1 -- 172.21.15.29:0/2508210489 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd2d400f690 con 0x7fd2d80ffe40 2024-01-24T13:35:33.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.582+0000 7fd2ce7fc700 1 -- 172.21.15.29:0/2508210489 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d4003a60 con 0x7fd2d80ffe40 2024-01-24T13:35:33.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.582+0000 7fd2df741700 1 -- 172.21.15.29:0/2508210489 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2d81a12f0 con 0x7fd2d80ffe40 2024-01-24T13:35:33.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.583+0000 7fd2df741700 1 -- 172.21.15.29:0/2508210489 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2d810b450 con 0x7fd2d80ffe40 2024-01-24T13:35:33.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.584+0000 7fd2ce7fc700 1 -- 172.21.15.29:0/2508210489 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7fd2d400fb00 con 0x7fd2d80ffe40 2024-01-24T13:35:33.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.585+0000 7fd2ce7fc700 1 --2- 172.21.15.29:0/2508210489 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fd2c40a0520 0x7fd2c40a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:33.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.585+0000 7fd2dccdc700 1 --2- 172.21.15.29:0/2508210489 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fd2c40a0520 0x7fd2c40a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:33.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.585+0000 7fd2ce7fc700 1 -- 172.21.15.29:0/2508210489 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8698+0+0 (secure 0 0 0) 0x7fd2d40cba60 con 0x7fd2d80ffe40 2024-01-24T13:35:33.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.585+0000 7fd2dccdc700 1 --2- 172.21.15.29:0/2508210489 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fd2c40a0520 0x7fd2c40a29e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8100f10 tx=0x7fd2c800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:33.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.590+0000 7fd2ce7fc700 1 -- 172.21.15.29:0/2508210489 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fd2d4095430 con 0x7fd2d80ffe40 2024-01-24T13:35:33.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:33 smithi163 ceph-mon[44969]: pgmap v223: 97 pgs: 16 creating+peering, 57 active+clean, 24 unknown; 578 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 2 op/s 2024-01-24T13:35:33.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:33 smithi163 ceph-mon[44969]: osdmap e82: 12 total, 12 up, 12 in 2024-01-24T13:35:33.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:33.799+0000 7fd2df741700 1 -- 172.21.15.29:0/2508210489 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7fd2d81a1b60 con 0x7fd2d80ffe40 2024-01-24T13:35:33.804 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:33 smithi187 ceph-mon[44911]: pgmap v223: 97 pgs: 16 creating+peering, 57 active+clean, 24 unknown; 578 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 2 op/s 2024-01-24T13:35:33.804 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:33 smithi187 ceph-mon[44911]: osdmap e82: 12 total, 12 up, 12 in 2024-01-24T13:35:33.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:33 smithi029 ceph-mon[40651]: pgmap v223: 97 pgs: 16 creating+peering, 57 active+clean, 24 unknown; 578 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 2 op/s 2024-01-24T13:35:33.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:33 smithi029 ceph-mon[40651]: osdmap e82: 12 total, 12 up, 12 in 2024-01-24T13:35:34.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.412+0000 7fd2ce7fc700 1 -- 172.21.15.29:0/2508210489 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v84) v1 ==== 169+0+0 (secure 0 0 0) 0x7fd2d4098a50 con 0x7fd2d80ffe40 2024-01-24T13:35:34.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.424+0000 7fd2df741700 1 -- 172.21.15.29:0/2508210489 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fd2c40a0520 msgr2=0x7fd2c40a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:34.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.424+0000 7fd2df741700 1 --2- 172.21.15.29:0/2508210489 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fd2c40a0520 0x7fd2c40a29e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8100f10 tx=0x7fd2c800b410 comp rx=0 tx=0).stop 2024-01-24T13:35:34.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.424+0000 7fd2df741700 1 -- 172.21.15.29:0/2508210489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2d80ffe40 msgr2=0x7fd2d8103bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:34.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.424+0000 7fd2df741700 1 --2- 172.21.15.29:0/2508210489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2d80ffe40 0x7fd2d8103bf0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd2d400d980 tx=0x7fd2d400dc90 comp rx=0 tx=0).stop 2024-01-24T13:35:34.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.431+0000 7fd2df741700 1 -- 172.21.15.29:0/2508210489 shutdown_connections 2024-01-24T13:35:34.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.431+0000 7fd2df741700 1 --2- 172.21.15.29:0/2508210489 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2d81011f0 0x7fd2d8102770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:34.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.431+0000 7fd2df741700 1 --2- 172.21.15.29:0/2508210489 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fd2c40a0520 0x7fd2c40a29e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:34.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.431+0000 7fd2df741700 1 --2- 172.21.15.29:0/2508210489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2d80ffe40 0x7fd2d8103bf0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:34.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.431+0000 7fd2df741700 1 --2- 172.21.15.29:0/2508210489 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd2d8100800 0x7fd2d8102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:34.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.431+0000 7fd2df741700 1 -- 172.21.15.29:0/2508210489 >> 172.21.15.29:0/2508210489 conn(0x7fd2d80fa910 msgr2=0x7fd2d80fed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:34.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.433+0000 7fd2df741700 1 -- 172.21.15.29:0/2508210489 shutdown_connections 2024-01-24T13:35:34.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.434+0000 7fd2df741700 1 -- 172.21.15.29:0/2508210489 wait complete. 2024-01-24T13:35:34.438 INFO:teuthology.orchestra.run.smithi029.stderr:set pool 4 allow_ec_overwrites to true 2024-01-24T13:35:34.451 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-01-24T13:35:34.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.591+0000 7f899c477700 1 -- 172.21.15.29:0/3317709336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89940ffc70 msgr2=0x7f8994100090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:34.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.591+0000 7f899c477700 1 --2- 172.21.15.29:0/3317709336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89940ffc70 0x7f8994100090 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f89900099c0 tx=0x7f8990009cd0 comp rx=0 tx=0).stop 2024-01-24T13:35:34.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.591+0000 7f899c477700 1 -- 172.21.15.29:0/3317709336 shutdown_connections 2024-01-24T13:35:34.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.591+0000 7f899c477700 1 --2- 172.21.15.29:0/3317709336 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8994101800 0x7f89941080b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:34.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.591+0000 7f899c477700 1 --2- 172.21.15.29:0/3317709336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89940ffc70 0x7f8994100090 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:34.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.591+0000 7f899c477700 1 --2- 172.21.15.29:0/3317709336 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8994100e60 0x7f89941012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:34.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.591+0000 7f899c477700 1 -- 172.21.15.29:0/3317709336 >> 172.21.15.29:0/3317709336 conn(0x7f89940fb200 msgr2=0x7f89940fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:34.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.591+0000 7f899c477700 1 -- 172.21.15.29:0/3317709336 shutdown_connections 2024-01-24T13:35:34.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.592+0000 7f899c477700 1 -- 172.21.15.29:0/3317709336 wait complete. 2024-01-24T13:35:34.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.592+0000 7f899c477700 1 Processor -- start 2024-01-24T13:35:34.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.592+0000 7f899c477700 1 -- start start 2024-01-24T13:35:34.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.592+0000 7f899c477700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89940ffc70 0x7f89940699c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:34.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.592+0000 7f899c477700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8994100e60 0x7f899419dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:34.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.592+0000 7f899c477700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8994101800 0x7f89941a0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:34.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.592+0000 7f899c477700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89941a07f0 con 0x7f89940ffc70 2024-01-24T13:35:34.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.592+0000 7f899c477700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89941a0960 con 0x7f8994100e60 2024-01-24T13:35:34.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.592+0000 7f899c477700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89941a0b20 con 0x7f8994101800 2024-01-24T13:35:34.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.592+0000 7f899a213700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89940ffc70 0x7f89940699c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:34.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.592+0000 7f899a213700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89940ffc70 0x7f89940699c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52298/0 (socket says 172.21.15.29:52298) 2024-01-24T13:35:34.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.593+0000 7f899a213700 1 -- 172.21.15.29:0/3100257544 learned_addr learned my addr 172.21.15.29:0/3100257544 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:34.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.593+0000 7f899aa14700 1 --2- 172.21.15.29:0/3100257544 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8994101800 0x7f89941a0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:34.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.593+0000 7f8999a12700 1 --2- 172.21.15.29:0/3100257544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8994100e60 0x7f899419dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:34.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.593+0000 7f899a213700 1 -- 172.21.15.29:0/3100257544 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8994101800 msgr2=0x7f89941a0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:34.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.593+0000 7f899a213700 1 --2- 172.21.15.29:0/3100257544 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8994101800 0x7f89941a0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:34.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.593+0000 7f899a213700 1 -- 172.21.15.29:0/3100257544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8994100e60 msgr2=0x7f899419dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:34.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.593+0000 7f899a213700 1 --2- 172.21.15.29:0/3100257544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8994100e60 0x7f899419dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:34.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.593+0000 7f899a213700 1 -- 172.21.15.29:0/3100257544 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89900095f0 con 0x7f89940ffc70 2024-01-24T13:35:34.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.593+0000 7f899a213700 1 --2- 172.21.15.29:0/3100257544 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89940ffc70 0x7f89940699c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f89900036f0 tx=0x7f8990009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:34.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.593+0000 7f898b7fe700 1 -- 172.21.15.29:0/3100257544 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f899001d070 con 0x7f89940ffc70 2024-01-24T13:35:34.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.593+0000 7f898b7fe700 1 -- 172.21.15.29:0/3100257544 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8990004010 con 0x7f89940ffc70 2024-01-24T13:35:34.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.593+0000 7f899c477700 1 -- 172.21.15.29:0/3100257544 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89941a0da0 con 0x7f89940ffc70 2024-01-24T13:35:34.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.593+0000 7f898b7fe700 1 -- 172.21.15.29:0/3100257544 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f899000fe70 con 0x7f89940ffc70 2024-01-24T13:35:34.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.593+0000 7f899c477700 1 -- 172.21.15.29:0/3100257544 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89941a11d0 con 0x7f89940ffc70 2024-01-24T13:35:34.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.594+0000 7f899c477700 1 -- 172.21.15.29:0/3100257544 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8994107260 con 0x7f89940ffc70 2024-01-24T13:35:34.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.596+0000 7f898b7fe700 1 -- 172.21.15.29:0/3100257544 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f899000f560 con 0x7f89940ffc70 2024-01-24T13:35:34.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.596+0000 7f898b7fe700 1 --2- 172.21.15.29:0/3100257544 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f89800a05a0 0x7f89800a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:34.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.596+0000 7f898b7fe700 1 -- 172.21.15.29:0/3100257544 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8698+0+0 (secure 0 0 0) 0x7f89900ccda0 con 0x7f89940ffc70 2024-01-24T13:35:34.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.596+0000 7f8999a12700 1 --2- 172.21.15.29:0/3100257544 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f89800a05a0 0x7f89800a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:34.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.597+0000 7f8999a12700 1 --2- 172.21.15.29:0/3100257544 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f89800a05a0 0x7f89800a2a60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8984005950 tx=0x7f89840058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:34.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.600+0000 7f898b7fe700 1 -- 172.21.15.29:0/3100257544 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f8990096800 con 0x7f89940ffc70 2024-01-24T13:35:34.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:34 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/1945596773' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-01-24T13:35:34.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:34 smithi163 ceph-mon[44969]: osdmap e83: 12 total, 12 up, 12 in 2024-01-24T13:35:34.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:34 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/2508210489' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-01-24T13:35:34.804 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:34 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/1945596773' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-01-24T13:35:34.804 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:34 smithi187 ceph-mon[44911]: osdmap e83: 12 total, 12 up, 12 in 2024-01-24T13:35:34.804 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:34 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/2508210489' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-01-24T13:35:34.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:34 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/1945596773' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-01-24T13:35:34.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:34 smithi029 ceph-mon[40651]: osdmap e83: 12 total, 12 up, 12 in 2024-01-24T13:35:34.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:34 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/2508210489' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-01-24T13:35:34.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.864+0000 7f899c477700 1 -- 172.21.15.29:0/3100257544 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7f899402cc70 con 0x7f89940ffc70 2024-01-24T13:35:34.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.870+0000 7f898b7fe700 1 -- 172.21.15.29:0/3100257544 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v16) v1 ==== 137+0+0 (secure 0 0 0) 0x7f8990099e20 con 0x7f89940ffc70 2024-01-24T13:35:34.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.873+0000 7f899c477700 1 -- 172.21.15.29:0/3100257544 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f89800a05a0 msgr2=0x7f89800a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:34.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.873+0000 7f899c477700 1 --2- 172.21.15.29:0/3100257544 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f89800a05a0 0x7f89800a2a60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8984005950 tx=0x7f89840058e0 comp rx=0 tx=0).stop 2024-01-24T13:35:34.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.873+0000 7f899c477700 1 -- 172.21.15.29:0/3100257544 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89940ffc70 msgr2=0x7f89940699c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:34.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.873+0000 7f899c477700 1 --2- 172.21.15.29:0/3100257544 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89940ffc70 0x7f89940699c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f89900036f0 tx=0x7f8990009580 comp rx=0 tx=0).stop 2024-01-24T13:35:34.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.873+0000 7f899c477700 1 -- 172.21.15.29:0/3100257544 shutdown_connections 2024-01-24T13:35:34.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.873+0000 7f899c477700 1 --2- 172.21.15.29:0/3100257544 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8994101800 0x7f89941a0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:34.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.873+0000 7f899c477700 1 --2- 172.21.15.29:0/3100257544 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f89800a05a0 0x7f89800a2a60 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:34.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.873+0000 7f899c477700 1 --2- 172.21.15.29:0/3100257544 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89940ffc70 0x7f89940699c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:34.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.873+0000 7f899c477700 1 --2- 172.21.15.29:0/3100257544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8994100e60 0x7f899419dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:34.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.873+0000 7f899c477700 1 -- 172.21.15.29:0/3100257544 >> 172.21.15.29:0/3100257544 conn(0x7f89940fb200 msgr2=0x7f89941043a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:34.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.873+0000 7f899c477700 1 -- 172.21.15.29:0/3100257544 shutdown_connections 2024-01-24T13:35:34.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:34.873+0000 7f899c477700 1 -- 172.21.15.29:0/3100257544 wait complete. 2024-01-24T13:35:34.875 INFO:teuthology.orchestra.run.smithi029.stderr:added data pool 4 to fsmap 2024-01-24T13:35:34.888 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T13:35:35.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.024+0000 7f3ed0950700 1 -- 172.21.15.29:0/1213694021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ecc1017f0 msgr2=0x7f3ecc1080a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:35.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.024+0000 7f3ed0950700 1 --2- 172.21.15.29:0/1213694021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ecc1017f0 0x7f3ecc1080a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3ec400b0d0 tx=0x7f3ec400b3e0 comp rx=0 tx=0).stop 2024-01-24T13:35:35.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.024+0000 7f3ed0950700 1 -- 172.21.15.29:0/1213694021 shutdown_connections 2024-01-24T13:35:35.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.024+0000 7f3ed0950700 1 --2- 172.21.15.29:0/1213694021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ecc1017f0 0x7f3ecc1080a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:35.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.024+0000 7f3ed0950700 1 --2- 172.21.15.29:0/1213694021 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ecc0ffc60 0x7f3ecc100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:35.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.024+0000 7f3ed0950700 1 --2- 172.21.15.29:0/1213694021 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3ecc100e50 0x7f3ecc1012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:35.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.024+0000 7f3ed0950700 1 -- 172.21.15.29:0/1213694021 >> 172.21.15.29:0/1213694021 conn(0x7f3ecc0fb1d0 msgr2=0x7f3ecc0fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:35.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.024+0000 7f3ed0950700 1 -- 172.21.15.29:0/1213694021 shutdown_connections 2024-01-24T13:35:35.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.024+0000 7f3ed0950700 1 -- 172.21.15.29:0/1213694021 wait complete. 2024-01-24T13:35:35.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.025+0000 7f3ed0950700 1 Processor -- start 2024-01-24T13:35:35.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.025+0000 7f3ed0950700 1 -- start start 2024-01-24T13:35:35.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.025+0000 7f3ed0950700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ecc0ffc60 0x7f3ecc103bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:35.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.025+0000 7f3ed0950700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3ecc100e50 0x7f3ecc102230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:35.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.025+0000 7f3ed0950700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ecc1017f0 0x7f3ecc102770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:35.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.025+0000 7f3ed0950700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ecc102f10 con 0x7f3ecc0ffc60 2024-01-24T13:35:35.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.025+0000 7f3ed0950700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ecc103080 con 0x7f3ecc100e50 2024-01-24T13:35:35.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.025+0000 7f3ed0950700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ecc1a0a90 con 0x7f3ecc1017f0 2024-01-24T13:35:35.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.025+0000 7f3eca59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ecc0ffc60 0x7f3ecc103bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:35.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.025+0000 7f3eca59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ecc0ffc60 0x7f3ecc103bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52314/0 (socket says 172.21.15.29:52314) 2024-01-24T13:35:35.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.025+0000 7f3eca59c700 1 -- 172.21.15.29:0/1216126053 learned_addr learned my addr 172.21.15.29:0/1216126053 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:35.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.025+0000 7f3ec9d9b700 1 --2- 172.21.15.29:0/1216126053 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3ecc100e50 0x7f3ecc102230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:35.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.025+0000 7f3ecad9d700 1 --2- 172.21.15.29:0/1216126053 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ecc1017f0 0x7f3ecc102770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:35.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.026+0000 7f3eca59c700 1 -- 172.21.15.29:0/1216126053 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ecc1017f0 msgr2=0x7f3ecc102770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:35.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.026+0000 7f3eca59c700 1 --2- 172.21.15.29:0/1216126053 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ecc1017f0 0x7f3ecc102770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:35.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.026+0000 7f3eca59c700 1 -- 172.21.15.29:0/1216126053 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3ecc100e50 msgr2=0x7f3ecc102230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:35.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.026+0000 7f3eca59c700 1 --2- 172.21.15.29:0/1216126053 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3ecc100e50 0x7f3ecc102230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:35.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.026+0000 7f3eca59c700 1 -- 172.21.15.29:0/1216126053 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec4009d00 con 0x7f3ecc0ffc60 2024-01-24T13:35:35.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.026+0000 7f3eca59c700 1 --2- 172.21.15.29:0/1216126053 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ecc0ffc60 0x7f3ecc103bf0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc00c990 tx=0x7f3ebc00cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:35.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.026+0000 7f3ec37fe700 1 -- 172.21.15.29:0/1216126053 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ebc007940 con 0x7f3ecc0ffc60 2024-01-24T13:35:35.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.026+0000 7f3ec37fe700 1 -- 172.21.15.29:0/1216126053 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3ebc00f450 con 0x7f3ecc0ffc60 2024-01-24T13:35:35.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.026+0000 7f3ed0950700 1 -- 172.21.15.29:0/1216126053 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ecc1a0d70 con 0x7f3ecc0ffc60 2024-01-24T13:35:35.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.026+0000 7f3ec37fe700 1 -- 172.21.15.29:0/1216126053 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ebc018660 con 0x7f3ecc0ffc60 2024-01-24T13:35:35.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.026+0000 7f3ed0950700 1 -- 172.21.15.29:0/1216126053 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ecc1a11d0 con 0x7f3ecc0ffc60 2024-01-24T13:35:35.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.027+0000 7f3ed0950700 1 -- 172.21.15.29:0/1216126053 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ecc10b480 con 0x7f3ecc0ffc60 2024-01-24T13:35:35.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.028+0000 7f3ec37fe700 1 -- 172.21.15.29:0/1216126053 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f3ebc00f8c0 con 0x7f3ecc0ffc60 2024-01-24T13:35:35.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.029+0000 7f3ec37fe700 1 --2- 172.21.15.29:0/1216126053 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f3eb80a0570 0x7f3eb80a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:35.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.029+0000 7f3ec9d9b700 1 --2- 172.21.15.29:0/1216126053 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f3eb80a0570 0x7f3eb80a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:35.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.029+0000 7f3ec37fe700 1 -- 172.21.15.29:0/1216126053 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f3ebc0cbb40 con 0x7f3ecc0ffc60 2024-01-24T13:35:35.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.030+0000 7f3ec9d9b700 1 --2- 172.21.15.29:0/1216126053 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f3eb80a0570 0x7f3eb80a2a30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3eb4005950 tx=0x7f3eb400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:35.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.033+0000 7f3ec37fe700 1 -- 172.21.15.29:0/1216126053 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f3ebc095580 con 0x7f3ecc0ffc60 2024-01-24T13:35:35.232 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jan 24 13:35:34 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-b[82918]: 2024-01-24T13:35:34.853+0000 7ff9d0b13700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-24T13:35:35.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.291+0000 7f3ed0950700 1 -- 172.21.15.29:0/1216126053 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ecc10b670 con 0x7f3ecc0ffc60 2024-01-24T13:35:35.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.292+0000 7f3ec37fe700 1 -- 172.21.15.29:0/1216126053 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+10813 (secure 0 0 0) 0x7f3ebc098ba0 con 0x7f3ecc0ffc60 2024-01-24T13:35:35.293 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:35:35.293 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1295649805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1295649805},{"type":"v1","addr":"172.21.15.29:6837","nonce":1295649805}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14559,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2665026288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2665026288},{"type":"v1","addr":"172.21.15.163:6835","nonce":2665026288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":14598,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6839/581075460","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":581075460},{"type":"v1","addr":"172.21.15.29:6839","nonce":581075460}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14610,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6841/146067403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":146067403},{"type":"v1","addr":"172.21.15.29:6841","nonce":146067403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3136190517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3136190517},{"type":"v1","addr":"172.21.15.163:6833","nonce":3136190517}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6833/3688255584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":3688255584},{"type":"v1","addr":"172.21.15.187:6833","nonce":3688255584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24469,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2401960181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2401960181},{"type":"v1","addr":"172.21.15.29:6835","nonce":2401960181}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/1709179808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1709179808},{"type":"v1","addr":"172.21.15.187:6835","nonce":1709179808}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/533724911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":533724911},{"type":"v1","addr":"172.21.15.187:6837","nonce":533724911}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/1610317406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":1610317406},{"type":"v1","addr":"172.21.15.163:6837","nonce":1610317406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24523,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6839/3209015343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6838","nonce":3209015343},{"type":"v1","addr":"172.21.15.187:6839","nonce":3209015343}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T13:35:30.391749+0000","modified":"2024-01-24T13:35:34.867038+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24548},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24548":{"gid":24548,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.163:6839/2641650917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":2641650917},{"type":"v1","addr":"172.21.15.163:6839","nonce":2641650917}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-01-24T13:35:35.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.295+0000 7f3ed0950700 1 -- 172.21.15.29:0/1216126053 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f3eb80a0570 msgr2=0x7f3eb80a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:35.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.295+0000 7f3ed0950700 1 --2- 172.21.15.29:0/1216126053 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f3eb80a0570 0x7f3eb80a2a30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3eb4005950 tx=0x7f3eb400a300 comp rx=0 tx=0).stop 2024-01-24T13:35:35.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.295+0000 7f3ed0950700 1 -- 172.21.15.29:0/1216126053 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ecc0ffc60 msgr2=0x7f3ecc103bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:35.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.295+0000 7f3ed0950700 1 --2- 172.21.15.29:0/1216126053 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ecc0ffc60 0x7f3ecc103bf0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc00c990 tx=0x7f3ebc00cca0 comp rx=0 tx=0).stop 2024-01-24T13:35:35.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.295+0000 7f3ed0950700 1 -- 172.21.15.29:0/1216126053 shutdown_connections 2024-01-24T13:35:35.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.295+0000 7f3ed0950700 1 --2- 172.21.15.29:0/1216126053 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ecc1017f0 0x7f3ecc102770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:35.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.295+0000 7f3ed0950700 1 --2- 172.21.15.29:0/1216126053 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f3eb80a0570 0x7f3eb80a2a30 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:35.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.295+0000 7f3ed0950700 1 --2- 172.21.15.29:0/1216126053 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ecc0ffc60 0x7f3ecc103bf0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:35.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.295+0000 7f3ed0950700 1 --2- 172.21.15.29:0/1216126053 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3ecc100e50 0x7f3ecc102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:35.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.295+0000 7f3ed0950700 1 -- 172.21.15.29:0/1216126053 >> 172.21.15.29:0/1216126053 conn(0x7f3ecc0fb1d0 msgr2=0x7f3ecc068f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:35.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.295+0000 7f3ed0950700 1 -- 172.21.15.29:0/1216126053 shutdown_connections 2024-01-24T13:35:35.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.295+0000 7f3ed0950700 1 -- 172.21.15.29:0/1216126053 wait complete. 2024-01-24T13:35:35.297 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 16 2024-01-24T13:35:35.310 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T13:35:35.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.447+0000 7f654bacd700 1 -- 172.21.15.29:0/1073538208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65440ffbb0 msgr2=0x7f65440fffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:35.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.447+0000 7f654bacd700 1 --2- 172.21.15.29:0/1073538208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65440ffbb0 0x7f65440fffd0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f65400099c0 tx=0x7f6540009cd0 comp rx=0 tx=0).stop 2024-01-24T13:35:35.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.447+0000 7f654bacd700 1 -- 172.21.15.29:0/1073538208 shutdown_connections 2024-01-24T13:35:35.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.447+0000 7f654bacd700 1 --2- 172.21.15.29:0/1073538208 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6544101740 0x7f6544107ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:35.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.447+0000 7f654bacd700 1 --2- 172.21.15.29:0/1073538208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65440ffbb0 0x7f65440fffd0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:35.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.447+0000 7f654bacd700 1 --2- 172.21.15.29:0/1073538208 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6544100da0 0x7f6544101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:35.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.447+0000 7f654bacd700 1 -- 172.21.15.29:0/1073538208 >> 172.21.15.29:0/1073538208 conn(0x7f65440fb100 msgr2=0x7f65440fd560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:35.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.447+0000 7f654bacd700 1 -- 172.21.15.29:0/1073538208 shutdown_connections 2024-01-24T13:35:35.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.447+0000 7f654bacd700 1 -- 172.21.15.29:0/1073538208 wait complete. 2024-01-24T13:35:35.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.448+0000 7f654bacd700 1 Processor -- start 2024-01-24T13:35:35.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.448+0000 7f654bacd700 1 -- start start 2024-01-24T13:35:35.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.448+0000 7f654bacd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65440ffbb0 0x7f6544068a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:35.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.448+0000 7f654bacd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6544100da0 0x7f654419b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:35.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.448+0000 7f654bacd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6544101740 0x7f654419dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:35.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.448+0000 7f654bacd700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f654419e4b0 con 0x7f65440ffbb0 2024-01-24T13:35:35.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.448+0000 7f654bacd700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f654419e620 con 0x7f6544101740 2024-01-24T13:35:35.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.448+0000 7f654bacd700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f654419e7e0 con 0x7f6544100da0 2024-01-24T13:35:35.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.448+0000 7f6549869700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65440ffbb0 0x7f6544068a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:35.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.448+0000 7f6549869700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65440ffbb0 0x7f6544068a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52338/0 (socket says 172.21.15.29:52338) 2024-01-24T13:35:35.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.448+0000 7f6549869700 1 -- 172.21.15.29:0/2170862591 learned_addr learned my addr 172.21.15.29:0/2170862591 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:35.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.448+0000 7f6549068700 1 --2- 172.21.15.29:0/2170862591 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6544100da0 0x7f654419b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:35.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.448+0000 7f654a06a700 1 --2- 172.21.15.29:0/2170862591 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6544101740 0x7f654419dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:35.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.448+0000 7f6549869700 1 -- 172.21.15.29:0/2170862591 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6544100da0 msgr2=0x7f654419b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:35.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.448+0000 7f6549869700 1 --2- 172.21.15.29:0/2170862591 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6544100da0 0x7f654419b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:35.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.448+0000 7f6549869700 1 -- 172.21.15.29:0/2170862591 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6544101740 msgr2=0x7f654419dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:35.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.448+0000 7f6549869700 1 --2- 172.21.15.29:0/2170862591 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6544101740 0x7f654419dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:35.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.448+0000 7f6549869700 1 -- 172.21.15.29:0/2170862591 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65400095f0 con 0x7f65440ffbb0 2024-01-24T13:35:35.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.448+0000 7f6549068700 1 --2- 172.21.15.29:0/2170862591 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6544100da0 0x7f654419b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T13:35:35.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.449+0000 7f6549869700 1 --2- 172.21.15.29:0/2170862591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65440ffbb0 0x7f6544068a30 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6540009990 tx=0x7f6540005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:35.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.449+0000 7f653affd700 1 -- 172.21.15.29:0/2170862591 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f654001d070 con 0x7f65440ffbb0 2024-01-24T13:35:35.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.449+0000 7f654bacd700 1 -- 172.21.15.29:0/2170862591 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f654419ea60 con 0x7f65440ffbb0 2024-01-24T13:35:35.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.449+0000 7f653affd700 1 -- 172.21.15.29:0/2170862591 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6540004200 con 0x7f65440ffbb0 2024-01-24T13:35:35.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.449+0000 7f653affd700 1 -- 172.21.15.29:0/2170862591 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6540022e30 con 0x7f65440ffbb0 2024-01-24T13:35:35.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.449+0000 7f654bacd700 1 -- 172.21.15.29:0/2170862591 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f654419ef60 con 0x7f65440ffbb0 2024-01-24T13:35:35.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.450+0000 7f654bacd700 1 -- 172.21.15.29:0/2170862591 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6544106b30 con 0x7f65440ffbb0 2024-01-24T13:35:35.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.451+0000 7f653affd700 1 -- 172.21.15.29:0/2170862591 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f6540003d40 con 0x7f65440ffbb0 2024-01-24T13:35:35.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.452+0000 7f653affd700 1 --2- 172.21.15.29:0/2170862591 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f65300a0570 0x7f65300a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:35.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.452+0000 7f653affd700 1 -- 172.21.15.29:0/2170862591 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f65400cce10 con 0x7f65440ffbb0 2024-01-24T13:35:35.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.452+0000 7f6549068700 1 --2- 172.21.15.29:0/2170862591 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f65300a0570 0x7f65300a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:35.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.452+0000 7f6549068700 1 --2- 172.21.15.29:0/2170862591 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f65300a0570 0x7f65300a2a30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6534005fd0 tx=0x7f6534005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:35.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.456+0000 7f653affd700 1 -- 172.21.15.29:0/2170862591 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f6540096850 con 0x7f65440ffbb0 2024-01-24T13:35:35.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:35 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/2508210489' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-01-24T13:35:35.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:35 smithi029 ceph-mon[40651]: osdmap e84: 12 total, 12 up, 12 in 2024-01-24T13:35:35.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:35 smithi029 ceph-mon[40651]: pgmap v226: 129 pgs: 16 creating+peering, 69 active+clean, 44 unknown; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 3 op/s 2024-01-24T13:35:35.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:35 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/3100257544' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-01-24T13:35:35.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:35 smithi029 ceph-mon[40651]: osdmap e85: 12 total, 12 up, 12 in 2024-01-24T13:35:35.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:35 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/3100257544' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-01-24T13:35:35.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:35 smithi029 ceph-mon[40651]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-01-24T13:35:35.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:35 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/1216126053' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T13:35:35.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.671+0000 7f654bacd700 1 -- 172.21.15.29:0/2170862591 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65440619f0 con 0x7f65440ffbb0 2024-01-24T13:35:35.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.673+0000 7f653affd700 1 -- 172.21.15.29:0/2170862591 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23981 (secure 0 0 0) 0x7f6540099e70 con 0x7f65440ffbb0 2024-01-24T13:35:35.674 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:35:35.674 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":85,"fsid":"83afcf5e-babc-11ee-95b1-87774f69a715","created":"2024-01-24T13:29:33.175895+0000","modified":"2024-01-24T13:35:34.866792+0000","last_up_change":"2024-01-24T13:34:57.141902+0000","last_in_change":"2024-01-24T13:34:41.071943+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T13:32:21.264634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-24T13:35:28.761733+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-24T13:35:29.763293+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-01-24T13:35:32.858844+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9857a607-2e6f-41a5-a86a-795970be6210","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":2406629076},{"type":"v1","addr":"172.21.15.29:6803","nonce":2406629076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":2406629076},{"type":"v1","addr":"172.21.15.29:6805","nonce":2406629076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2406629076},{"type":"v1","addr":"172.21.15.29:6809","nonce":2406629076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":2406629076},{"type":"v1","addr":"172.21.15.29:6807","nonce":2406629076}]},"public_addr":"172.21.15.29:6803/2406629076","cluster_addr":"172.21.15.29:6805/2406629076","heartbeat_back_addr":"172.21.15.29:6809/2406629076","heartbeat_front_addr":"172.21.15.29:6807/2406629076","state":["exists","up"]},{"osd":1,"uuid":"8bb2ba48-23d4-4bbd-a912-ae74f9e7f6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":2217244566},{"type":"v1","addr":"172.21.15.163:6801","nonce":2217244566}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":2217244566},{"type":"v1","addr":"172.21.15.163:6803","nonce":2217244566}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":2217244566},{"type":"v1","addr":"172.21.15.163:6807","nonce":2217244566}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":2217244566},{"type":"v1","addr":"172.21.15.163:6805","nonce":2217244566}]},"public_addr":"172.21.15.163:6801/2217244566","cluster_addr":"172.21.15.163:6803/2217244566","heartbeat_back_addr":"172.21.15.163:6807/2217244566","heartbeat_front_addr":"172.21.15.163:6805/2217244566","state":["exists","up"]},{"osd":2,"uuid":"f0ed4131-cd29-4706-89ea-fde40f46ea83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3589694992},{"type":"v1","addr":"172.21.15.187:6801","nonce":3589694992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3589694992},{"type":"v1","addr":"172.21.15.187:6803","nonce":3589694992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3589694992},{"type":"v1","addr":"172.21.15.187:6807","nonce":3589694992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3589694992},{"type":"v1","addr":"172.21.15.187:6805","nonce":3589694992}]},"public_addr":"172.21.15.187:6801/3589694992","cluster_addr":"172.21.15.187:6803/3589694992","heartbeat_back_addr":"172.21.15.187:6807/3589694992","heartbeat_front_addr":"172.21.15.187:6805/3589694992","state":["exists","up"]},{"osd":3,"uuid":"da821310-b502-4b77-a5ce-0ffed0a3dafc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3417210415},{"type":"v1","addr":"172.21.15.29:6811","nonce":3417210415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":3417210415},{"type":"v1","addr":"172.21.15.29:6813","nonce":3417210415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3417210415},{"type":"v1","addr":"172.21.15.29:6817","nonce":3417210415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3417210415},{"type":"v1","addr":"172.21.15.29:6815","nonce":3417210415}]},"public_addr":"172.21.15.29:6811/3417210415","cluster_addr":"172.21.15.29:6813/3417210415","heartbeat_back_addr":"172.21.15.29:6817/3417210415","heartbeat_front_addr":"172.21.15.29:6815/3417210415","state":["exists","up"]},{"osd":4,"uuid":"8b3a348f-e7dd-4ccb-82a5-2d3f5d101da8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3283787183},{"type":"v1","addr":"172.21.15.163:6809","nonce":3283787183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":3283787183},{"type":"v1","addr":"172.21.15.163:6811","nonce":3283787183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3283787183},{"type":"v1","addr":"172.21.15.163:6815","nonce":3283787183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":3283787183},{"type":"v1","addr":"172.21.15.163:6813","nonce":3283787183}]},"public_addr":"172.21.15.163:6809/3283787183","cluster_addr":"172.21.15.163:6811/3283787183","heartbeat_back_addr":"172.21.15.163:6815/3283787183","heartbeat_front_addr":"172.21.15.163:6813/3283787183","state":["exists","up"]},{"osd":5,"uuid":"c26c7155-618b-4712-acdb-d4fd6c8b7f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":2259228817},{"type":"v1","addr":"172.21.15.187:6809","nonce":2259228817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":2259228817},{"type":"v1","addr":"172.21.15.187:6811","nonce":2259228817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":2259228817},{"type":"v1","addr":"172.21.15.187:6815","nonce":2259228817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":2259228817},{"type":"v1","addr":"172.21.15.187:6813","nonce":2259228817}]},"public_addr":"172.21.15.187:6809/2259228817","cluster_addr":"172.21.15.187:6811/2259228817","heartbeat_back_addr":"172.21.15.187:6815/2259228817","heartbeat_front_addr":"172.21.15.187:6813/2259228817","state":["exists","up"]},{"osd":6,"uuid":"042ac0b4-1fe1-444f-bf51-ad89613a2185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":938400701},{"type":"v1","addr":"172.21.15.29:6819","nonce":938400701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":938400701},{"type":"v1","addr":"172.21.15.29:6821","nonce":938400701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":938400701},{"type":"v1","addr":"172.21.15.29:6825","nonce":938400701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":938400701},{"type":"v1","addr":"172.21.15.29:6823","nonce":938400701}]},"public_addr":"172.21.15.29:6819/938400701","cluster_addr":"172.21.15.29:6821/938400701","heartbeat_back_addr":"172.21.15.29:6825/938400701","heartbeat_front_addr":"172.21.15.29:6823/938400701","state":["exists","up"]},{"osd":7,"uuid":"add3d09b-76f3-4a93-83a2-30ac94c3fad1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2783976092},{"type":"v1","addr":"172.21.15.163:6817","nonce":2783976092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2783976092},{"type":"v1","addr":"172.21.15.163:6819","nonce":2783976092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2783976092},{"type":"v1","addr":"172.21.15.163:6823","nonce":2783976092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2783976092},{"type":"v1","addr":"172.21.15.163:6821","nonce":2783976092}]},"public_addr":"172.21.15.163:6817/2783976092","cluster_addr":"172.21.15.163:6819/2783976092","heartbeat_back_addr":"172.21.15.163:6823/2783976092","heartbeat_front_addr":"172.21.15.163:6821/2783976092","state":["exists","up"]},{"osd":8,"uuid":"71044d6a-b55f-4bcf-94c5-79f71eb7b422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":2856803882},{"type":"v1","addr":"172.21.15.187:6817","nonce":2856803882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":2856803882},{"type":"v1","addr":"172.21.15.187:6819","nonce":2856803882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":2856803882},{"type":"v1","addr":"172.21.15.187:6823","nonce":2856803882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":2856803882},{"type":"v1","addr":"172.21.15.187:6821","nonce":2856803882}]},"public_addr":"172.21.15.187:6817/2856803882","cluster_addr":"172.21.15.187:6819/2856803882","heartbeat_back_addr":"172.21.15.187:6823/2856803882","heartbeat_front_addr":"172.21.15.187:6821/2856803882","state":["exists","up"]},{"osd":9,"uuid":"cf417a8e-e576-43de-b89c-f104ee3969ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1392813017},{"type":"v1","addr":"172.21.15.29:6827","nonce":1392813017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1392813017},{"type":"v1","addr":"172.21.15.29:6829","nonce":1392813017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1392813017},{"type":"v1","addr":"172.21.15.29:6833","nonce":1392813017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1392813017},{"type":"v1","addr":"172.21.15.29:6831","nonce":1392813017}]},"public_addr":"172.21.15.29:6827/1392813017","cluster_addr":"172.21.15.29:6829/1392813017","heartbeat_back_addr":"172.21.15.29:6833/1392813017","heartbeat_front_addr":"172.21.15.29:6831/1392813017","state":["exists","up"]},{"osd":10,"uuid":"bc7cfb9d-3bfa-45b2-929a-a8dc90f1a2a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":2523735721},{"type":"v1","addr":"172.21.15.163:6825","nonce":2523735721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":2523735721},{"type":"v1","addr":"172.21.15.163:6827","nonce":2523735721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":2523735721},{"type":"v1","addr":"172.21.15.163:6831","nonce":2523735721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":2523735721},{"type":"v1","addr":"172.21.15.163:6829","nonce":2523735721}]},"public_addr":"172.21.15.163:6825/2523735721","cluster_addr":"172.21.15.163:6827/2523735721","heartbeat_back_addr":"172.21.15.163:6831/2523735721","heartbeat_front_addr":"172.21.15.163:6829/2523735721","state":["exists","up"]},{"osd":11,"uuid":"1453684e-fe11-4cbc-a868-cf90b3a7a021","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1828122314},{"type":"v1","addr":"172.21.15.187:6825","nonce":1828122314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1828122314},{"type":"v1","addr":"172.21.15.187:6827","nonce":1828122314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":1828122314},{"type":"v1","addr":"172.21.15.187:6831","nonce":1828122314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":1828122314},{"type":"v1","addr":"172.21.15.187:6829","nonce":1828122314}]},"public_addr":"172.21.15.187:6825/1828122314","cluster_addr":"172.21.15.187:6827/1828122314","heartbeat_back_addr":"172.21.15.187:6831/1828122314","heartbeat_front_addr":"172.21.15.187:6829/1828122314","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:31:43.572543+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:32:00.759082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:32:17.705002+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:32:35.666788+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:32:53.221701+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:33:10.290988+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:33:27.946743+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:33:45.698074+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:34:02.750761+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:34:20.662278+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:34:38.233389+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:34:55.419540+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/1419474286":"2024-01-25T13:30:30.728638+0000","172.21.15.29:6800/835547951":"2024-01-25T13:30:08.775587+0000","172.21.15.29:6801/1996568496":"2024-01-25T13:30:30.728638+0000","172.21.15.29:0/2727527401":"2024-01-25T13:30:08.775587+0000","172.21.15.29:0/41509684":"2024-01-25T13:30:30.728638+0000","172.21.15.29:6800/1996568496":"2024-01-25T13:30:30.728638+0000","172.21.15.29:0/2833864599":"2024-01-25T13:30:30.728638+0000","172.21.15.29:0/168349482":"2024-01-25T13:30:08.775587+0000","172.21.15.29:6801/835547951":"2024-01-25T13:30:08.775587+0000","172.21.15.29:0/1142561996":"2024-01-25T13:30:08.775587+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T13:35:35.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.675+0000 7f654bacd700 1 -- 172.21.15.29:0/2170862591 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f65300a0570 msgr2=0x7f65300a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:35.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.675+0000 7f654bacd700 1 --2- 172.21.15.29:0/2170862591 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f65300a0570 0x7f65300a2a30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6534005fd0 tx=0x7f6534005f60 comp rx=0 tx=0).stop 2024-01-24T13:35:35.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.675+0000 7f654bacd700 1 -- 172.21.15.29:0/2170862591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65440ffbb0 msgr2=0x7f6544068a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:35.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.675+0000 7f654bacd700 1 --2- 172.21.15.29:0/2170862591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65440ffbb0 0x7f6544068a30 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6540009990 tx=0x7f6540005e80 comp rx=0 tx=0).stop 2024-01-24T13:35:35.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.675+0000 7f654bacd700 1 -- 172.21.15.29:0/2170862591 shutdown_connections 2024-01-24T13:35:35.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.675+0000 7f654bacd700 1 --2- 172.21.15.29:0/2170862591 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6544101740 0x7f654419dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:35.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.675+0000 7f654bacd700 1 --2- 172.21.15.29:0/2170862591 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f65300a0570 0x7f65300a2a30 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:35.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.675+0000 7f654bacd700 1 --2- 172.21.15.29:0/2170862591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65440ffbb0 0x7f6544068a30 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:35.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.675+0000 7f654bacd700 1 --2- 172.21.15.29:0/2170862591 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6544100da0 0x7f654419b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:35.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.675+0000 7f654bacd700 1 -- 172.21.15.29:0/2170862591 >> 172.21.15.29:0/2170862591 conn(0x7f65440fb100 msgr2=0x7f65440fd410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:35.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.675+0000 7f654bacd700 1 -- 172.21.15.29:0/2170862591 shutdown_connections 2024-01-24T13:35:35.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.675+0000 7f654bacd700 1 -- 172.21.15.29:0/2170862591 wait complete. 2024-01-24T13:35:35.691 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T13:35:35.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:35 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/2508210489' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-01-24T13:35:35.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:35 smithi163 ceph-mon[44969]: osdmap e84: 12 total, 12 up, 12 in 2024-01-24T13:35:35.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:35 smithi163 ceph-mon[44969]: pgmap v226: 129 pgs: 16 creating+peering, 69 active+clean, 44 unknown; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 3 op/s 2024-01-24T13:35:35.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:35 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/3100257544' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-01-24T13:35:35.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:35 smithi163 ceph-mon[44969]: osdmap e85: 12 total, 12 up, 12 in 2024-01-24T13:35:35.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:35 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/3100257544' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-01-24T13:35:35.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:35 smithi163 ceph-mon[44969]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-01-24T13:35:35.733 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:35 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/1216126053' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T13:35:35.804 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:35 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/2508210489' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-01-24T13:35:35.804 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:35 smithi187 ceph-mon[44911]: osdmap e84: 12 total, 12 up, 12 in 2024-01-24T13:35:35.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:35 smithi187 ceph-mon[44911]: pgmap v226: 129 pgs: 16 creating+peering, 69 active+clean, 44 unknown; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 3 op/s 2024-01-24T13:35:35.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:35 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/3100257544' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-01-24T13:35:35.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:35 smithi187 ceph-mon[44911]: osdmap e85: 12 total, 12 up, 12 in 2024-01-24T13:35:35.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:35 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/3100257544' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-01-24T13:35:35.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:35 smithi187 ceph-mon[44911]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-01-24T13:35:35.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:35 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/1216126053' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T13:35:35.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.828+0000 7f6dc5e40700 1 -- 172.21.15.29:0/660037038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dc01028e0 msgr2=0x7f6dc0109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:35.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.828+0000 7f6dc5e40700 1 --2- 172.21.15.29:0/660037038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dc01028e0 0x7f6dc0109190 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6db800b0d0 tx=0x7f6db800b3e0 comp rx=0 tx=0).stop 2024-01-24T13:35:35.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.828+0000 7f6dc5e40700 1 -- 172.21.15.29:0/660037038 shutdown_connections 2024-01-24T13:35:35.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.828+0000 7f6dc5e40700 1 --2- 172.21.15.29:0/660037038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dc01028e0 0x7f6dc0109190 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:35.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.828+0000 7f6dc5e40700 1 --2- 172.21.15.29:0/660037038 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6dc0100d50 0x7f6dc0101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:35.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.828+0000 7f6dc5e40700 1 --2- 172.21.15.29:0/660037038 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6dc0101f40 0x7f6dc01023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:35.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.828+0000 7f6dc5e40700 1 -- 172.21.15.29:0/660037038 >> 172.21.15.29:0/660037038 conn(0x7f6dc00fc280 msgr2=0x7f6dc00fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:35.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.828+0000 7f6dc5e40700 1 -- 172.21.15.29:0/660037038 shutdown_connections 2024-01-24T13:35:35.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.828+0000 7f6dc5e40700 1 -- 172.21.15.29:0/660037038 wait complete. 2024-01-24T13:35:35.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.829+0000 7f6dc5e40700 1 Processor -- start 2024-01-24T13:35:35.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.829+0000 7f6dc5e40700 1 -- start start 2024-01-24T13:35:35.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.829+0000 7f6dc5e40700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6dc0100d50 0x7f6dc0196ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:35.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.829+0000 7f6dc5e40700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6dc0101f40 0x7f6dc0196ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:35.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.829+0000 7f6dc5e40700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dc01028e0 0x7f6dc019e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:35.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.829+0000 7f6dc5e40700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc019e630 con 0x7f6dc01028e0 2024-01-24T13:35:35.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.829+0000 7f6dc5e40700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc019e770 con 0x7f6dc0101f40 2024-01-24T13:35:35.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.829+0000 7f6dc5e40700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc019e900 con 0x7f6dc0100d50 2024-01-24T13:35:35.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.829+0000 7f6dbffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dc01028e0 0x7f6dc019e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:35.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.829+0000 7f6dbffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dc01028e0 0x7f6dc019e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52360/0 (socket says 172.21.15.29:52360) 2024-01-24T13:35:35.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.829+0000 7f6dbffff700 1 -- 172.21.15.29:0/2582547028 learned_addr learned my addr 172.21.15.29:0/2582547028 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:35.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.829+0000 7f6dbf7fe700 1 --2- 172.21.15.29:0/2582547028 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6dc0100d50 0x7f6dc0196ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:35.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.829+0000 7f6dbeffd700 1 --2- 172.21.15.29:0/2582547028 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6dc0101f40 0x7f6dc0196ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:35.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.830+0000 7f6dbffff700 1 -- 172.21.15.29:0/2582547028 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6dc0100d50 msgr2=0x7f6dc0196ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:35.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.830+0000 7f6dbffff700 1 --2- 172.21.15.29:0/2582547028 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6dc0100d50 0x7f6dc0196ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:35.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.830+0000 7f6dbffff700 1 -- 172.21.15.29:0/2582547028 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6dc0101f40 msgr2=0x7f6dc0196ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:35.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.830+0000 7f6dbffff700 1 --2- 172.21.15.29:0/2582547028 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6dc0101f40 0x7f6dc0196ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:35.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.830+0000 7f6dbffff700 1 -- 172.21.15.29:0/2582547028 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db8009d00 con 0x7f6dc01028e0 2024-01-24T13:35:35.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.830+0000 7f6dbf7fe700 1 --2- 172.21.15.29:0/2582547028 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6dc0100d50 0x7f6dc0196ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T13:35:35.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.830+0000 7f6dbeffd700 1 --2- 172.21.15.29:0/2582547028 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6dc0101f40 0x7f6dc0196ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T13:35:35.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.830+0000 7f6dbffff700 1 --2- 172.21.15.29:0/2582547028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dc01028e0 0x7f6dc019e060 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6db8008f10 tx=0x7f6db8008ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:35.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.831+0000 7f6dbcff9700 1 -- 172.21.15.29:0/2582547028 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db800e040 con 0x7f6dc01028e0 2024-01-24T13:35:35.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.831+0000 7f6dbcff9700 1 -- 172.21.15.29:0/2582547028 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6db8003c00 con 0x7f6dc01028e0 2024-01-24T13:35:35.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.831+0000 7f6dbcff9700 1 -- 172.21.15.29:0/2582547028 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db801b7d0 con 0x7f6dc01028e0 2024-01-24T13:35:35.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.831+0000 7f6dc5e40700 1 -- 172.21.15.29:0/2582547028 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dc019eb80 con 0x7f6dc01028e0 2024-01-24T13:35:35.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.831+0000 7f6dc5e40700 1 -- 172.21.15.29:0/2582547028 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dc019ef50 con 0x7f6dc01028e0 2024-01-24T13:35:35.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.832+0000 7f6dc5e40700 1 -- 172.21.15.29:0/2582547028 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dc01081e0 con 0x7f6dc01028e0 2024-01-24T13:35:35.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.833+0000 7f6dbcff9700 1 -- 172.21.15.29:0/2582547028 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f6db8005090 con 0x7f6dc01028e0 2024-01-24T13:35:35.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.834+0000 7f6dbcff9700 1 --2- 172.21.15.29:0/2582547028 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f6da80a0570 0x7f6da80a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:35.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.834+0000 7f6dbcff9700 1 -- 172.21.15.29:0/2582547028 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f6db80cd650 con 0x7f6dc01028e0 2024-01-24T13:35:35.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.834+0000 7f6dbf7fe700 1 --2- 172.21.15.29:0/2582547028 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f6da80a0570 0x7f6da80a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:35.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.834+0000 7f6dbf7fe700 1 --2- 172.21.15.29:0/2582547028 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f6da80a0570 0x7f6da80a2a30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6dc0101da0 tx=0x7f6db0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:35.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:35.840+0000 7f6dbcff9700 1 -- 172.21.15.29:0/2582547028 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f6db8097090 con 0x7f6dc01028e0 2024-01-24T13:35:36.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.051+0000 7f6dc5e40700 1 -- 172.21.15.29:0/2582547028 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6dc002ce40 con 0x7f6dc01028e0 2024-01-24T13:35:36.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.053+0000 7f6dbcff9700 1 -- 172.21.15.29:0/2582547028 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23981 (secure 0 0 0) 0x7f6db809a6b0 con 0x7f6dc01028e0 2024-01-24T13:35:36.054 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:35:36.054 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":85,"fsid":"83afcf5e-babc-11ee-95b1-87774f69a715","created":"2024-01-24T13:29:33.175895+0000","modified":"2024-01-24T13:35:34.866792+0000","last_up_change":"2024-01-24T13:34:57.141902+0000","last_in_change":"2024-01-24T13:34:41.071943+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T13:32:21.264634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-24T13:35:28.761733+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-24T13:35:29.763293+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-01-24T13:35:32.858844+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9857a607-2e6f-41a5-a86a-795970be6210","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":2406629076},{"type":"v1","addr":"172.21.15.29:6803","nonce":2406629076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":2406629076},{"type":"v1","addr":"172.21.15.29:6805","nonce":2406629076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2406629076},{"type":"v1","addr":"172.21.15.29:6809","nonce":2406629076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":2406629076},{"type":"v1","addr":"172.21.15.29:6807","nonce":2406629076}]},"public_addr":"172.21.15.29:6803/2406629076","cluster_addr":"172.21.15.29:6805/2406629076","heartbeat_back_addr":"172.21.15.29:6809/2406629076","heartbeat_front_addr":"172.21.15.29:6807/2406629076","state":["exists","up"]},{"osd":1,"uuid":"8bb2ba48-23d4-4bbd-a912-ae74f9e7f6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":2217244566},{"type":"v1","addr":"172.21.15.163:6801","nonce":2217244566}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":2217244566},{"type":"v1","addr":"172.21.15.163:6803","nonce":2217244566}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":2217244566},{"type":"v1","addr":"172.21.15.163:6807","nonce":2217244566}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":2217244566},{"type":"v1","addr":"172.21.15.163:6805","nonce":2217244566}]},"public_addr":"172.21.15.163:6801/2217244566","cluster_addr":"172.21.15.163:6803/2217244566","heartbeat_back_addr":"172.21.15.163:6807/2217244566","heartbeat_front_addr":"172.21.15.163:6805/2217244566","state":["exists","up"]},{"osd":2,"uuid":"f0ed4131-cd29-4706-89ea-fde40f46ea83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3589694992},{"type":"v1","addr":"172.21.15.187:6801","nonce":3589694992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3589694992},{"type":"v1","addr":"172.21.15.187:6803","nonce":3589694992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3589694992},{"type":"v1","addr":"172.21.15.187:6807","nonce":3589694992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3589694992},{"type":"v1","addr":"172.21.15.187:6805","nonce":3589694992}]},"public_addr":"172.21.15.187:6801/3589694992","cluster_addr":"172.21.15.187:6803/3589694992","heartbeat_back_addr":"172.21.15.187:6807/3589694992","heartbeat_front_addr":"172.21.15.187:6805/3589694992","state":["exists","up"]},{"osd":3,"uuid":"da821310-b502-4b77-a5ce-0ffed0a3dafc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3417210415},{"type":"v1","addr":"172.21.15.29:6811","nonce":3417210415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":3417210415},{"type":"v1","addr":"172.21.15.29:6813","nonce":3417210415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3417210415},{"type":"v1","addr":"172.21.15.29:6817","nonce":3417210415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3417210415},{"type":"v1","addr":"172.21.15.29:6815","nonce":3417210415}]},"public_addr":"172.21.15.29:6811/3417210415","cluster_addr":"172.21.15.29:6813/3417210415","heartbeat_back_addr":"172.21.15.29:6817/3417210415","heartbeat_front_addr":"172.21.15.29:6815/3417210415","state":["exists","up"]},{"osd":4,"uuid":"8b3a348f-e7dd-4ccb-82a5-2d3f5d101da8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3283787183},{"type":"v1","addr":"172.21.15.163:6809","nonce":3283787183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":3283787183},{"type":"v1","addr":"172.21.15.163:6811","nonce":3283787183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3283787183},{"type":"v1","addr":"172.21.15.163:6815","nonce":3283787183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":3283787183},{"type":"v1","addr":"172.21.15.163:6813","nonce":3283787183}]},"public_addr":"172.21.15.163:6809/3283787183","cluster_addr":"172.21.15.163:6811/3283787183","heartbeat_back_addr":"172.21.15.163:6815/3283787183","heartbeat_front_addr":"172.21.15.163:6813/3283787183","state":["exists","up"]},{"osd":5,"uuid":"c26c7155-618b-4712-acdb-d4fd6c8b7f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":2259228817},{"type":"v1","addr":"172.21.15.187:6809","nonce":2259228817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":2259228817},{"type":"v1","addr":"172.21.15.187:6811","nonce":2259228817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":2259228817},{"type":"v1","addr":"172.21.15.187:6815","nonce":2259228817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":2259228817},{"type":"v1","addr":"172.21.15.187:6813","nonce":2259228817}]},"public_addr":"172.21.15.187:6809/2259228817","cluster_addr":"172.21.15.187:6811/2259228817","heartbeat_back_addr":"172.21.15.187:6815/2259228817","heartbeat_front_addr":"172.21.15.187:6813/2259228817","state":["exists","up"]},{"osd":6,"uuid":"042ac0b4-1fe1-444f-bf51-ad89613a2185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":938400701},{"type":"v1","addr":"172.21.15.29:6819","nonce":938400701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":938400701},{"type":"v1","addr":"172.21.15.29:6821","nonce":938400701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":938400701},{"type":"v1","addr":"172.21.15.29:6825","nonce":938400701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":938400701},{"type":"v1","addr":"172.21.15.29:6823","nonce":938400701}]},"public_addr":"172.21.15.29:6819/938400701","cluster_addr":"172.21.15.29:6821/938400701","heartbeat_back_addr":"172.21.15.29:6825/938400701","heartbeat_front_addr":"172.21.15.29:6823/938400701","state":["exists","up"]},{"osd":7,"uuid":"add3d09b-76f3-4a93-83a2-30ac94c3fad1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2783976092},{"type":"v1","addr":"172.21.15.163:6817","nonce":2783976092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2783976092},{"type":"v1","addr":"172.21.15.163:6819","nonce":2783976092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2783976092},{"type":"v1","addr":"172.21.15.163:6823","nonce":2783976092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2783976092},{"type":"v1","addr":"172.21.15.163:6821","nonce":2783976092}]},"public_addr":"172.21.15.163:6817/2783976092","cluster_addr":"172.21.15.163:6819/2783976092","heartbeat_back_addr":"172.21.15.163:6823/2783976092","heartbeat_front_addr":"172.21.15.163:6821/2783976092","state":["exists","up"]},{"osd":8,"uuid":"71044d6a-b55f-4bcf-94c5-79f71eb7b422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":2856803882},{"type":"v1","addr":"172.21.15.187:6817","nonce":2856803882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":2856803882},{"type":"v1","addr":"172.21.15.187:6819","nonce":2856803882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":2856803882},{"type":"v1","addr":"172.21.15.187:6823","nonce":2856803882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":2856803882},{"type":"v1","addr":"172.21.15.187:6821","nonce":2856803882}]},"public_addr":"172.21.15.187:6817/2856803882","cluster_addr":"172.21.15.187:6819/2856803882","heartbeat_back_addr":"172.21.15.187:6823/2856803882","heartbeat_front_addr":"172.21.15.187:6821/2856803882","state":["exists","up"]},{"osd":9,"uuid":"cf417a8e-e576-43de-b89c-f104ee3969ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1392813017},{"type":"v1","addr":"172.21.15.29:6827","nonce":1392813017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1392813017},{"type":"v1","addr":"172.21.15.29:6829","nonce":1392813017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1392813017},{"type":"v1","addr":"172.21.15.29:6833","nonce":1392813017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1392813017},{"type":"v1","addr":"172.21.15.29:6831","nonce":1392813017}]},"public_addr":"172.21.15.29:6827/1392813017","cluster_addr":"172.21.15.29:6829/1392813017","heartbeat_back_addr":"172.21.15.29:6833/1392813017","heartbeat_front_addr":"172.21.15.29:6831/1392813017","state":["exists","up"]},{"osd":10,"uuid":"bc7cfb9d-3bfa-45b2-929a-a8dc90f1a2a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":2523735721},{"type":"v1","addr":"172.21.15.163:6825","nonce":2523735721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":2523735721},{"type":"v1","addr":"172.21.15.163:6827","nonce":2523735721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":2523735721},{"type":"v1","addr":"172.21.15.163:6831","nonce":2523735721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":2523735721},{"type":"v1","addr":"172.21.15.163:6829","nonce":2523735721}]},"public_addr":"172.21.15.163:6825/2523735721","cluster_addr":"172.21.15.163:6827/2523735721","heartbeat_back_addr":"172.21.15.163:6831/2523735721","heartbeat_front_addr":"172.21.15.163:6829/2523735721","state":["exists","up"]},{"osd":11,"uuid":"1453684e-fe11-4cbc-a868-cf90b3a7a021","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1828122314},{"type":"v1","addr":"172.21.15.187:6825","nonce":1828122314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1828122314},{"type":"v1","addr":"172.21.15.187:6827","nonce":1828122314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":1828122314},{"type":"v1","addr":"172.21.15.187:6831","nonce":1828122314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":1828122314},{"type":"v1","addr":"172.21.15.187:6829","nonce":1828122314}]},"public_addr":"172.21.15.187:6825/1828122314","cluster_addr":"172.21.15.187:6827/1828122314","heartbeat_back_addr":"172.21.15.187:6831/1828122314","heartbeat_front_addr":"172.21.15.187:6829/1828122314","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:31:43.572543+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:32:00.759082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:32:17.705002+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:32:35.666788+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:32:53.221701+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:33:10.290988+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:33:27.946743+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:33:45.698074+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:34:02.750761+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:34:20.662278+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:34:38.233389+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:34:55.419540+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/1419474286":"2024-01-25T13:30:30.728638+0000","172.21.15.29:6800/835547951":"2024-01-25T13:30:08.775587+0000","172.21.15.29:6801/1996568496":"2024-01-25T13:30:30.728638+0000","172.21.15.29:0/2727527401":"2024-01-25T13:30:08.775587+0000","172.21.15.29:0/41509684":"2024-01-25T13:30:30.728638+0000","172.21.15.29:6800/1996568496":"2024-01-25T13:30:30.728638+0000","172.21.15.29:0/2833864599":"2024-01-25T13:30:30.728638+0000","172.21.15.29:0/168349482":"2024-01-25T13:30:08.775587+0000","172.21.15.29:6801/835547951":"2024-01-25T13:30:08.775587+0000","172.21.15.29:0/1142561996":"2024-01-25T13:30:08.775587+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T13:35:36.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.056+0000 7f6dc5e40700 1 -- 172.21.15.29:0/2582547028 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f6da80a0570 msgr2=0x7f6da80a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:36.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.056+0000 7f6dc5e40700 1 --2- 172.21.15.29:0/2582547028 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f6da80a0570 0x7f6da80a2a30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6dc0101da0 tx=0x7f6db0009260 comp rx=0 tx=0).stop 2024-01-24T13:35:36.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.056+0000 7f6dc5e40700 1 -- 172.21.15.29:0/2582547028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dc01028e0 msgr2=0x7f6dc019e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:36.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.056+0000 7f6dc5e40700 1 --2- 172.21.15.29:0/2582547028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dc01028e0 0x7f6dc019e060 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6db8008f10 tx=0x7f6db8008ff0 comp rx=0 tx=0).stop 2024-01-24T13:35:36.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.056+0000 7f6dc5e40700 1 -- 172.21.15.29:0/2582547028 shutdown_connections 2024-01-24T13:35:36.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.056+0000 7f6dc5e40700 1 --2- 172.21.15.29:0/2582547028 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f6da80a0570 0x7f6da80a2a30 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:36.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.056+0000 7f6dc5e40700 1 --2- 172.21.15.29:0/2582547028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dc01028e0 0x7f6dc019e060 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:36.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.056+0000 7f6dc5e40700 1 --2- 172.21.15.29:0/2582547028 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6dc0100d50 0x7f6dc0196ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:36.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.056+0000 7f6dc5e40700 1 --2- 172.21.15.29:0/2582547028 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6dc0101f40 0x7f6dc0196ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:36.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.056+0000 7f6dc5e40700 1 -- 172.21.15.29:0/2582547028 >> 172.21.15.29:0/2582547028 conn(0x7f6dc00fc280 msgr2=0x7f6dc0105320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:36.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.056+0000 7f6dc5e40700 1 -- 172.21.15.29:0/2582547028 shutdown_connections 2024-01-24T13:35:36.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.056+0000 7f6dc5e40700 1 -- 172.21.15.29:0/2582547028 wait complete. 2024-01-24T13:35:36.092 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T13:35:36.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.229+0000 7f80084ae700 1 -- 172.21.15.29:0/2164218970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80001017c0 msgr2=0x7f8000108070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:36.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.229+0000 7f80084ae700 1 --2- 172.21.15.29:0/2164218970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80001017c0 0x7f8000108070 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7ff800b0d0 tx=0x7f7ff800b3e0 comp rx=0 tx=0).stop 2024-01-24T13:35:36.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.230+0000 7f80084ae700 1 -- 172.21.15.29:0/2164218970 shutdown_connections 2024-01-24T13:35:36.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.230+0000 7f80084ae700 1 --2- 172.21.15.29:0/2164218970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80001017c0 0x7f8000108070 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:36.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.230+0000 7f80084ae700 1 --2- 172.21.15.29:0/2164218970 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80000ffc30 0x7f8000100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:36.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.230+0000 7f80084ae700 1 --2- 172.21.15.29:0/2164218970 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8000100e20 0x7f8000101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:36.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.230+0000 7f80084ae700 1 -- 172.21.15.29:0/2164218970 >> 172.21.15.29:0/2164218970 conn(0x7f80000fb1a0 msgr2=0x7f80000fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:36.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.230+0000 7f80084ae700 1 -- 172.21.15.29:0/2164218970 shutdown_connections 2024-01-24T13:35:36.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.230+0000 7f80084ae700 1 -- 172.21.15.29:0/2164218970 wait complete. 2024-01-24T13:35:36.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.230+0000 7f80084ae700 1 Processor -- start 2024-01-24T13:35:36.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.230+0000 7f80084ae700 1 -- start start 2024-01-24T13:35:36.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.231+0000 7f80084ae700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80000ffc30 0x7f8000196ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:36.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.231+0000 7f80084ae700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8000100e20 0x7f8000197020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:36.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.231+0000 7f80084ae700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80001017c0 0x7f800019e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:36.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.231+0000 7f80084ae700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800019e660 con 0x7f80001017c0 2024-01-24T13:35:36.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.231+0000 7f80084ae700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800019e7a0 con 0x7f8000100e20 2024-01-24T13:35:36.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.231+0000 7f80084ae700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800019e930 con 0x7f80000ffc30 2024-01-24T13:35:36.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.231+0000 7f8006a4b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80001017c0 0x7f800019e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:36.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.231+0000 7f8006a4b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80001017c0 0x7f800019e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52376/0 (socket says 172.21.15.29:52376) 2024-01-24T13:35:36.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.231+0000 7f8006a4b700 1 -- 172.21.15.29:0/1227071579 learned_addr learned my addr 172.21.15.29:0/1227071579 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:36.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.231+0000 7f8005a49700 1 --2- 172.21.15.29:0/1227071579 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8000100e20 0x7f8000197020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:36.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.231+0000 7f800624a700 1 --2- 172.21.15.29:0/1227071579 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80000ffc30 0x7f8000196ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:36.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.231+0000 7f8006a4b700 1 -- 172.21.15.29:0/1227071579 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80000ffc30 msgr2=0x7f8000196ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:36.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.231+0000 7f8006a4b700 1 --2- 172.21.15.29:0/1227071579 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80000ffc30 0x7f8000196ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:36.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.231+0000 7f8006a4b700 1 -- 172.21.15.29:0/1227071579 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8000100e20 msgr2=0x7f8000197020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:36.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.231+0000 7f8006a4b700 1 --2- 172.21.15.29:0/1227071579 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8000100e20 0x7f8000197020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:36.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.231+0000 7f8006a4b700 1 -- 172.21.15.29:0/1227071579 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff8009d00 con 0x7f80001017c0 2024-01-24T13:35:36.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.231+0000 7f8006a4b700 1 --2- 172.21.15.29:0/1227071579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80001017c0 0x7f800019e090 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7ff8008eb0 tx=0x7f7ff8008ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:36.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.231+0000 7f7ff77fe700 1 -- 172.21.15.29:0/1227071579 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff800e040 con 0x7f80001017c0 2024-01-24T13:35:36.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.231+0000 7f7ff77fe700 1 -- 172.21.15.29:0/1227071579 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ff800bdb0 con 0x7f80001017c0 2024-01-24T13:35:36.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.231+0000 7f80084ae700 1 -- 172.21.15.29:0/1227071579 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f800019ebb0 con 0x7f80001017c0 2024-01-24T13:35:36.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.232+0000 7f7ff77fe700 1 -- 172.21.15.29:0/1227071579 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff801d470 con 0x7f80001017c0 2024-01-24T13:35:36.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.232+0000 7f80084ae700 1 -- 172.21.15.29:0/1227071579 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f800019f030 con 0x7f80001017c0 2024-01-24T13:35:36.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.232+0000 7f80084ae700 1 -- 172.21.15.29:0/1227071579 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f800010b4a0 con 0x7f80001017c0 2024-01-24T13:35:36.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.234+0000 7f7ff77fe700 1 -- 172.21.15.29:0/1227071579 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f7ff8005000 con 0x7f80001017c0 2024-01-24T13:35:36.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.234+0000 7f7ff77fe700 1 --2- 172.21.15.29:0/1227071579 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f7fec0a05f0 0x7f7fec0a2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:36.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.235+0000 7f800624a700 1 --2- 172.21.15.29:0/1227071579 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f7fec0a05f0 0x7f7fec0a2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:36.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.235+0000 7f7ff77fe700 1 -- 172.21.15.29:0/1227071579 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f7ff80cd4b0 con 0x7f80001017c0 2024-01-24T13:35:36.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.235+0000 7f800624a700 1 --2- 172.21.15.29:0/1227071579 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f7fec0a05f0 0x7f7fec0a2ab0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8000100c80 tx=0x7f7ffc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:36.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.238+0000 7f7ff77fe700 1 -- 172.21.15.29:0/1227071579 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f7ff8096ef0 con 0x7f80001017c0 2024-01-24T13:35:36.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.500+0000 7f80084ae700 1 -- 172.21.15.29:0/1227071579 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f800019f5e0 con 0x7f80001017c0 2024-01-24T13:35:36.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.501+0000 7f7ff77fe700 1 -- 172.21.15.29:0/1227071579 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+10813 (secure 0 0 0) 0x7f7ff809a510 con 0x7f80001017c0 2024-01-24T13:35:36.502 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:35:36.502 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1295649805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1295649805},{"type":"v1","addr":"172.21.15.29:6837","nonce":1295649805}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14559,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2665026288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2665026288},{"type":"v1","addr":"172.21.15.163:6835","nonce":2665026288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":14598,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6839/581075460","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":581075460},{"type":"v1","addr":"172.21.15.29:6839","nonce":581075460}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14610,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6841/146067403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":146067403},{"type":"v1","addr":"172.21.15.29:6841","nonce":146067403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3136190517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3136190517},{"type":"v1","addr":"172.21.15.163:6833","nonce":3136190517}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6833/3688255584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":3688255584},{"type":"v1","addr":"172.21.15.187:6833","nonce":3688255584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24469,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2401960181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2401960181},{"type":"v1","addr":"172.21.15.29:6835","nonce":2401960181}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/1709179808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1709179808},{"type":"v1","addr":"172.21.15.187:6835","nonce":1709179808}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/533724911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":533724911},{"type":"v1","addr":"172.21.15.187:6837","nonce":533724911}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/1610317406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":1610317406},{"type":"v1","addr":"172.21.15.163:6837","nonce":1610317406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24523,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6839/3209015343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6838","nonce":3209015343},{"type":"v1","addr":"172.21.15.187:6839","nonce":3209015343}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T13:35:30.391749+0000","modified":"2024-01-24T13:35:34.867038+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24548},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24548":{"gid":24548,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.163:6839/2641650917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":2641650917},{"type":"v1","addr":"172.21.15.163:6839","nonce":2641650917}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-01-24T13:35:36.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.503+0000 7f80084ae700 1 -- 172.21.15.29:0/1227071579 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f7fec0a05f0 msgr2=0x7f7fec0a2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:36.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.503+0000 7f80084ae700 1 --2- 172.21.15.29:0/1227071579 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f7fec0a05f0 0x7f7fec0a2ab0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8000100c80 tx=0x7f7ffc009260 comp rx=0 tx=0).stop 2024-01-24T13:35:36.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.503+0000 7f80084ae700 1 -- 172.21.15.29:0/1227071579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80001017c0 msgr2=0x7f800019e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:36.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.503+0000 7f80084ae700 1 --2- 172.21.15.29:0/1227071579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80001017c0 0x7f800019e090 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7ff8008eb0 tx=0x7f7ff8008ee0 comp rx=0 tx=0).stop 2024-01-24T13:35:36.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.503+0000 7f80084ae700 1 -- 172.21.15.29:0/1227071579 shutdown_connections 2024-01-24T13:35:36.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.503+0000 7f80084ae700 1 --2- 172.21.15.29:0/1227071579 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f7fec0a05f0 0x7f7fec0a2ab0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:36.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.503+0000 7f80084ae700 1 --2- 172.21.15.29:0/1227071579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80001017c0 0x7f800019e090 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:36.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.503+0000 7f80084ae700 1 --2- 172.21.15.29:0/1227071579 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80000ffc30 0x7f8000196ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:36.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.503+0000 7f80084ae700 1 --2- 172.21.15.29:0/1227071579 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8000100e20 0x7f8000197020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:36.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.503+0000 7f80084ae700 1 -- 172.21.15.29:0/1227071579 >> 172.21.15.29:0/1227071579 conn(0x7f80000fb1a0 msgr2=0x7f8000069120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:36.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.503+0000 7f80084ae700 1 -- 172.21.15.29:0/1227071579 shutdown_connections 2024-01-24T13:35:36.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.503+0000 7f80084ae700 1 -- 172.21.15.29:0/1227071579 wait complete. 2024-01-24T13:35:36.505 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 16 2024-01-24T13:35:36.518 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T13:35:36.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.655+0000 7f80aba7b700 1 -- 172.21.15.29:0/2260743617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80a4100e10 msgr2=0x7f80a4101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:36.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.655+0000 7f80aba7b700 1 --2- 172.21.15.29:0/2260743617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80a4100e10 0x7f80a4101270 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f80940099c0 tx=0x7f8094009cd0 comp rx=0 tx=0).stop 2024-01-24T13:35:36.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.655+0000 7f80aba7b700 1 -- 172.21.15.29:0/2260743617 shutdown_connections 2024-01-24T13:35:36.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.655+0000 7f80aba7b700 1 --2- 172.21.15.29:0/2260743617 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80a41017b0 0x7f80a4108060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:36.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.655+0000 7f80aba7b700 1 --2- 172.21.15.29:0/2260743617 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80a40ffc20 0x7f80a4100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:36.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.655+0000 7f80aba7b700 1 --2- 172.21.15.29:0/2260743617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80a4100e10 0x7f80a4101270 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:36.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.655+0000 7f80aba7b700 1 -- 172.21.15.29:0/2260743617 >> 172.21.15.29:0/2260743617 conn(0x7f80a40fb170 msgr2=0x7f80a40fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:36.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.655+0000 7f80aba7b700 1 -- 172.21.15.29:0/2260743617 shutdown_connections 2024-01-24T13:35:36.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.655+0000 7f80aba7b700 1 -- 172.21.15.29:0/2260743617 wait complete. 2024-01-24T13:35:36.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.655+0000 7f80aba7b700 1 Processor -- start 2024-01-24T13:35:36.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.656+0000 7f80aba7b700 1 -- start start 2024-01-24T13:35:36.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.656+0000 7f80aba7b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80a40ffc20 0x7f80a4068a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:36.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.656+0000 7f80aba7b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80a4100e10 0x7f80a419b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:36.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.656+0000 7f80aba7b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80a41017b0 0x7f80a419dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:36.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.656+0000 7f80aba7b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a419e520 con 0x7f80a40ffc20 2024-01-24T13:35:36.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.656+0000 7f80aba7b700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a419e690 con 0x7f80a41017b0 2024-01-24T13:35:36.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.656+0000 7f80aba7b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a419e850 con 0x7f80a4100e10 2024-01-24T13:35:36.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.656+0000 7f80a9817700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80a40ffc20 0x7f80a4068a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:36.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.656+0000 7f80a9817700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80a40ffc20 0x7f80a4068a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52390/0 (socket says 172.21.15.29:52390) 2024-01-24T13:35:36.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.656+0000 7f80a9817700 1 -- 172.21.15.29:0/1791887298 learned_addr learned my addr 172.21.15.29:0/1791887298 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:36.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.656+0000 7f80a9016700 1 --2- 172.21.15.29:0/1791887298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80a4100e10 0x7f80a419b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:36.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.656+0000 7f80aa018700 1 --2- 172.21.15.29:0/1791887298 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80a41017b0 0x7f80a419dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:36.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.656+0000 7f80a9817700 1 -- 172.21.15.29:0/1791887298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80a4100e10 msgr2=0x7f80a419b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:36.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.656+0000 7f80a9817700 1 --2- 172.21.15.29:0/1791887298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80a4100e10 0x7f80a419b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:36.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.656+0000 7f80a9817700 1 -- 172.21.15.29:0/1791887298 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80a41017b0 msgr2=0x7f80a419dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:36.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.656+0000 7f80a9817700 1 --2- 172.21.15.29:0/1791887298 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80a41017b0 0x7f80a419dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:36.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.656+0000 7f80a9817700 1 -- 172.21.15.29:0/1791887298 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80940095f0 con 0x7f80a40ffc20 2024-01-24T13:35:36.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.656+0000 7f80aa018700 1 --2- 172.21.15.29:0/1791887298 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80a41017b0 0x7f80a419dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T13:35:36.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.656+0000 7f80a9817700 1 --2- 172.21.15.29:0/1791887298 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80a40ffc20 0x7f80a4068a30 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f80a000e970 tx=0x7f80a000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:36.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.657+0000 7f809affd700 1 -- 172.21.15.29:0/1791887298 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a0009790 con 0x7f80a40ffc20 2024-01-24T13:35:36.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.657+0000 7f809affd700 1 -- 172.21.15.29:0/1791887298 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f80a000eec0 con 0x7f80a40ffc20 2024-01-24T13:35:36.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.657+0000 7f80aba7b700 1 -- 172.21.15.29:0/1791887298 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80a419eb30 con 0x7f80a40ffc20 2024-01-24T13:35:36.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.657+0000 7f809affd700 1 -- 172.21.15.29:0/1791887298 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a0003cf0 con 0x7f80a40ffc20 2024-01-24T13:35:36.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.657+0000 7f80aba7b700 1 -- 172.21.15.29:0/1791887298 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80a419f060 con 0x7f80a40ffc20 2024-01-24T13:35:36.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.658+0000 7f80aba7b700 1 -- 172.21.15.29:0/1791887298 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80a410b430 con 0x7f80a40ffc20 2024-01-24T13:35:36.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.659+0000 7f809affd700 1 -- 172.21.15.29:0/1791887298 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f80a000ccd0 con 0x7f80a40ffc20 2024-01-24T13:35:36.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.660+0000 7f809affd700 1 --2- 172.21.15.29:0/1791887298 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f80900a05f0 0x7f80900a2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:36.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.660+0000 7f80a9016700 1 --2- 172.21.15.29:0/1791887298 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f80900a05f0 0x7f80900a2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:36.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.660+0000 7f809affd700 1 -- 172.21.15.29:0/1791887298 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f80a0014070 con 0x7f80a40ffc20 2024-01-24T13:35:36.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.661+0000 7f80a9016700 1 --2- 172.21.15.29:0/1791887298 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f80900a05f0 0x7f80900a2ab0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f809400b580 tx=0x7f8094005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:36.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.664+0000 7f809affd700 1 -- 172.21.15.29:0/1791887298 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f80a00958a0 con 0x7f80a40ffc20 2024-01-24T13:35:36.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:36 smithi163 ceph-mon[44969]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-01-24T13:35:36.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:36 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/2170862591' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T13:35:36.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:36 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/2582547028' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T13:35:36.804 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:36 smithi187 ceph-mon[44911]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-01-24T13:35:36.804 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:36 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/2170862591' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T13:35:36.804 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:36 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/2582547028' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T13:35:36.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:36 smithi029 ceph-mon[40651]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-01-24T13:35:36.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:36 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/2170862591' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T13:35:36.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:36 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/2582547028' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T13:35:36.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.877+0000 7f80aba7b700 1 -- 172.21.15.29:0/1791887298 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80a410b620 con 0x7f80a40ffc20 2024-01-24T13:35:36.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.878+0000 7f809affd700 1 -- 172.21.15.29:0/1791887298 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23981 (secure 0 0 0) 0x7f80a0095030 con 0x7f80a40ffc20 2024-01-24T13:35:36.880 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:35:36.880 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":85,"fsid":"83afcf5e-babc-11ee-95b1-87774f69a715","created":"2024-01-24T13:29:33.175895+0000","modified":"2024-01-24T13:35:34.866792+0000","last_up_change":"2024-01-24T13:34:57.141902+0000","last_in_change":"2024-01-24T13:34:41.071943+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T13:32:21.264634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-24T13:35:28.761733+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-24T13:35:29.763293+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-01-24T13:35:32.858844+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9857a607-2e6f-41a5-a86a-795970be6210","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":2406629076},{"type":"v1","addr":"172.21.15.29:6803","nonce":2406629076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":2406629076},{"type":"v1","addr":"172.21.15.29:6805","nonce":2406629076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2406629076},{"type":"v1","addr":"172.21.15.29:6809","nonce":2406629076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":2406629076},{"type":"v1","addr":"172.21.15.29:6807","nonce":2406629076}]},"public_addr":"172.21.15.29:6803/2406629076","cluster_addr":"172.21.15.29:6805/2406629076","heartbeat_back_addr":"172.21.15.29:6809/2406629076","heartbeat_front_addr":"172.21.15.29:6807/2406629076","state":["exists","up"]},{"osd":1,"uuid":"8bb2ba48-23d4-4bbd-a912-ae74f9e7f6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":2217244566},{"type":"v1","addr":"172.21.15.163:6801","nonce":2217244566}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":2217244566},{"type":"v1","addr":"172.21.15.163:6803","nonce":2217244566}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":2217244566},{"type":"v1","addr":"172.21.15.163:6807","nonce":2217244566}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":2217244566},{"type":"v1","addr":"172.21.15.163:6805","nonce":2217244566}]},"public_addr":"172.21.15.163:6801/2217244566","cluster_addr":"172.21.15.163:6803/2217244566","heartbeat_back_addr":"172.21.15.163:6807/2217244566","heartbeat_front_addr":"172.21.15.163:6805/2217244566","state":["exists","up"]},{"osd":2,"uuid":"f0ed4131-cd29-4706-89ea-fde40f46ea83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3589694992},{"type":"v1","addr":"172.21.15.187:6801","nonce":3589694992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3589694992},{"type":"v1","addr":"172.21.15.187:6803","nonce":3589694992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3589694992},{"type":"v1","addr":"172.21.15.187:6807","nonce":3589694992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3589694992},{"type":"v1","addr":"172.21.15.187:6805","nonce":3589694992}]},"public_addr":"172.21.15.187:6801/3589694992","cluster_addr":"172.21.15.187:6803/3589694992","heartbeat_back_addr":"172.21.15.187:6807/3589694992","heartbeat_front_addr":"172.21.15.187:6805/3589694992","state":["exists","up"]},{"osd":3,"uuid":"da821310-b502-4b77-a5ce-0ffed0a3dafc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3417210415},{"type":"v1","addr":"172.21.15.29:6811","nonce":3417210415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":3417210415},{"type":"v1","addr":"172.21.15.29:6813","nonce":3417210415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3417210415},{"type":"v1","addr":"172.21.15.29:6817","nonce":3417210415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3417210415},{"type":"v1","addr":"172.21.15.29:6815","nonce":3417210415}]},"public_addr":"172.21.15.29:6811/3417210415","cluster_addr":"172.21.15.29:6813/3417210415","heartbeat_back_addr":"172.21.15.29:6817/3417210415","heartbeat_front_addr":"172.21.15.29:6815/3417210415","state":["exists","up"]},{"osd":4,"uuid":"8b3a348f-e7dd-4ccb-82a5-2d3f5d101da8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3283787183},{"type":"v1","addr":"172.21.15.163:6809","nonce":3283787183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":3283787183},{"type":"v1","addr":"172.21.15.163:6811","nonce":3283787183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3283787183},{"type":"v1","addr":"172.21.15.163:6815","nonce":3283787183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":3283787183},{"type":"v1","addr":"172.21.15.163:6813","nonce":3283787183}]},"public_addr":"172.21.15.163:6809/3283787183","cluster_addr":"172.21.15.163:6811/3283787183","heartbeat_back_addr":"172.21.15.163:6815/3283787183","heartbeat_front_addr":"172.21.15.163:6813/3283787183","state":["exists","up"]},{"osd":5,"uuid":"c26c7155-618b-4712-acdb-d4fd6c8b7f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":2259228817},{"type":"v1","addr":"172.21.15.187:6809","nonce":2259228817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":2259228817},{"type":"v1","addr":"172.21.15.187:6811","nonce":2259228817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":2259228817},{"type":"v1","addr":"172.21.15.187:6815","nonce":2259228817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":2259228817},{"type":"v1","addr":"172.21.15.187:6813","nonce":2259228817}]},"public_addr":"172.21.15.187:6809/2259228817","cluster_addr":"172.21.15.187:6811/2259228817","heartbeat_back_addr":"172.21.15.187:6815/2259228817","heartbeat_front_addr":"172.21.15.187:6813/2259228817","state":["exists","up"]},{"osd":6,"uuid":"042ac0b4-1fe1-444f-bf51-ad89613a2185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":938400701},{"type":"v1","addr":"172.21.15.29:6819","nonce":938400701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":938400701},{"type":"v1","addr":"172.21.15.29:6821","nonce":938400701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":938400701},{"type":"v1","addr":"172.21.15.29:6825","nonce":938400701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":938400701},{"type":"v1","addr":"172.21.15.29:6823","nonce":938400701}]},"public_addr":"172.21.15.29:6819/938400701","cluster_addr":"172.21.15.29:6821/938400701","heartbeat_back_addr":"172.21.15.29:6825/938400701","heartbeat_front_addr":"172.21.15.29:6823/938400701","state":["exists","up"]},{"osd":7,"uuid":"add3d09b-76f3-4a93-83a2-30ac94c3fad1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2783976092},{"type":"v1","addr":"172.21.15.163:6817","nonce":2783976092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2783976092},{"type":"v1","addr":"172.21.15.163:6819","nonce":2783976092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2783976092},{"type":"v1","addr":"172.21.15.163:6823","nonce":2783976092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2783976092},{"type":"v1","addr":"172.21.15.163:6821","nonce":2783976092}]},"public_addr":"172.21.15.163:6817/2783976092","cluster_addr":"172.21.15.163:6819/2783976092","heartbeat_back_addr":"172.21.15.163:6823/2783976092","heartbeat_front_addr":"172.21.15.163:6821/2783976092","state":["exists","up"]},{"osd":8,"uuid":"71044d6a-b55f-4bcf-94c5-79f71eb7b422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":2856803882},{"type":"v1","addr":"172.21.15.187:6817","nonce":2856803882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":2856803882},{"type":"v1","addr":"172.21.15.187:6819","nonce":2856803882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":2856803882},{"type":"v1","addr":"172.21.15.187:6823","nonce":2856803882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":2856803882},{"type":"v1","addr":"172.21.15.187:6821","nonce":2856803882}]},"public_addr":"172.21.15.187:6817/2856803882","cluster_addr":"172.21.15.187:6819/2856803882","heartbeat_back_addr":"172.21.15.187:6823/2856803882","heartbeat_front_addr":"172.21.15.187:6821/2856803882","state":["exists","up"]},{"osd":9,"uuid":"cf417a8e-e576-43de-b89c-f104ee3969ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1392813017},{"type":"v1","addr":"172.21.15.29:6827","nonce":1392813017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1392813017},{"type":"v1","addr":"172.21.15.29:6829","nonce":1392813017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1392813017},{"type":"v1","addr":"172.21.15.29:6833","nonce":1392813017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1392813017},{"type":"v1","addr":"172.21.15.29:6831","nonce":1392813017}]},"public_addr":"172.21.15.29:6827/1392813017","cluster_addr":"172.21.15.29:6829/1392813017","heartbeat_back_addr":"172.21.15.29:6833/1392813017","heartbeat_front_addr":"172.21.15.29:6831/1392813017","state":["exists","up"]},{"osd":10,"uuid":"bc7cfb9d-3bfa-45b2-929a-a8dc90f1a2a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":2523735721},{"type":"v1","addr":"172.21.15.163:6825","nonce":2523735721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":2523735721},{"type":"v1","addr":"172.21.15.163:6827","nonce":2523735721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":2523735721},{"type":"v1","addr":"172.21.15.163:6831","nonce":2523735721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":2523735721},{"type":"v1","addr":"172.21.15.163:6829","nonce":2523735721}]},"public_addr":"172.21.15.163:6825/2523735721","cluster_addr":"172.21.15.163:6827/2523735721","heartbeat_back_addr":"172.21.15.163:6831/2523735721","heartbeat_front_addr":"172.21.15.163:6829/2523735721","state":["exists","up"]},{"osd":11,"uuid":"1453684e-fe11-4cbc-a868-cf90b3a7a021","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1828122314},{"type":"v1","addr":"172.21.15.187:6825","nonce":1828122314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1828122314},{"type":"v1","addr":"172.21.15.187:6827","nonce":1828122314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":1828122314},{"type":"v1","addr":"172.21.15.187:6831","nonce":1828122314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":1828122314},{"type":"v1","addr":"172.21.15.187:6829","nonce":1828122314}]},"public_addr":"172.21.15.187:6825/1828122314","cluster_addr":"172.21.15.187:6827/1828122314","heartbeat_back_addr":"172.21.15.187:6831/1828122314","heartbeat_front_addr":"172.21.15.187:6829/1828122314","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:31:43.572543+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:32:00.759082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:32:17.705002+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:32:35.666788+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:32:53.221701+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:33:10.290988+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:33:27.946743+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:33:45.698074+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:34:02.750761+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:34:20.662278+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:34:38.233389+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:34:55.419540+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/1419474286":"2024-01-25T13:30:30.728638+0000","172.21.15.29:6800/835547951":"2024-01-25T13:30:08.775587+0000","172.21.15.29:6801/1996568496":"2024-01-25T13:30:30.728638+0000","172.21.15.29:0/2727527401":"2024-01-25T13:30:08.775587+0000","172.21.15.29:0/41509684":"2024-01-25T13:30:30.728638+0000","172.21.15.29:6800/1996568496":"2024-01-25T13:30:30.728638+0000","172.21.15.29:0/2833864599":"2024-01-25T13:30:30.728638+0000","172.21.15.29:0/168349482":"2024-01-25T13:30:08.775587+0000","172.21.15.29:6801/835547951":"2024-01-25T13:30:08.775587+0000","172.21.15.29:0/1142561996":"2024-01-25T13:30:08.775587+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T13:35:36.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.881+0000 7f80aba7b700 1 -- 172.21.15.29:0/1791887298 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f80900a05f0 msgr2=0x7f80900a2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:36.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.881+0000 7f80aba7b700 1 --2- 172.21.15.29:0/1791887298 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f80900a05f0 0x7f80900a2ab0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f809400b580 tx=0x7f8094005fb0 comp rx=0 tx=0).stop 2024-01-24T13:35:36.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.881+0000 7f80aba7b700 1 -- 172.21.15.29:0/1791887298 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80a40ffc20 msgr2=0x7f80a4068a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:36.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.881+0000 7f80aba7b700 1 --2- 172.21.15.29:0/1791887298 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80a40ffc20 0x7f80a4068a30 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f80a000e970 tx=0x7f80a000ec80 comp rx=0 tx=0).stop 2024-01-24T13:35:36.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.881+0000 7f80aba7b700 1 -- 172.21.15.29:0/1791887298 shutdown_connections 2024-01-24T13:35:36.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.881+0000 7f80aba7b700 1 --2- 172.21.15.29:0/1791887298 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80a41017b0 0x7f80a419dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:36.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.881+0000 7f80aba7b700 1 --2- 172.21.15.29:0/1791887298 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f80900a05f0 0x7f80900a2ab0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:36.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.881+0000 7f80aba7b700 1 --2- 172.21.15.29:0/1791887298 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80a40ffc20 0x7f80a4068a30 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:36.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.881+0000 7f80aba7b700 1 --2- 172.21.15.29:0/1791887298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80a4100e10 0x7f80a419b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:36.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.881+0000 7f80aba7b700 1 -- 172.21.15.29:0/1791887298 >> 172.21.15.29:0/1791887298 conn(0x7f80a40fb170 msgr2=0x7f80a40fd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:36.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.881+0000 7f80aba7b700 1 -- 172.21.15.29:0/1791887298 shutdown_connections 2024-01-24T13:35:36.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:36.881+0000 7f80aba7b700 1 -- 172.21.15.29:0/1791887298 wait complete. 2024-01-24T13:35:36.897 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-24T13:35:30.391749+0000', 'modified': '2024-01-24T13:35:34.867038+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24548}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24548': {'gid': 24548, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.163:6839/2641650917', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6838', 'nonce': 2641650917}, {'type': 'v1', 'addr': '172.21.15.163:6839', 'nonce': 2641650917}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2024-01-24T13:35:36.897 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-24T13:35:36.897 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-01-24T13:35:37.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.035+0000 7f418beb2700 1 -- 172.21.15.29:0/3988081689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4184100e20 msgr2=0x7f4184101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.035+0000 7f418beb2700 1 --2- 172.21.15.29:0/3988081689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4184100e20 0x7f4184101280 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f41740099c0 tx=0x7f4174009cd0 comp rx=0 tx=0).stop 2024-01-24T13:35:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.036+0000 7f418beb2700 1 -- 172.21.15.29:0/3988081689 shutdown_connections 2024-01-24T13:35:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.036+0000 7f418beb2700 1 --2- 172.21.15.29:0/3988081689 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41841017c0 0x7f4184108070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.036+0000 7f418beb2700 1 --2- 172.21.15.29:0/3988081689 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41840ffc30 0x7f4184100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.036+0000 7f418beb2700 1 --2- 172.21.15.29:0/3988081689 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4184100e20 0x7f4184101280 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.036+0000 7f418beb2700 1 -- 172.21.15.29:0/3988081689 >> 172.21.15.29:0/3988081689 conn(0x7f41840fb1a0 msgr2=0x7f41840fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.036+0000 7f418beb2700 1 -- 172.21.15.29:0/3988081689 shutdown_connections 2024-01-24T13:35:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.036+0000 7f418beb2700 1 -- 172.21.15.29:0/3988081689 wait complete. 2024-01-24T13:35:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.036+0000 7f418beb2700 1 Processor -- start 2024-01-24T13:35:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.036+0000 7f418beb2700 1 -- start start 2024-01-24T13:35:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.036+0000 7f418beb2700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41840ffc30 0x7f41841969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.036+0000 7f418beb2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4184100e20 0x7f4184196f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.037+0000 7f418beb2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41841017c0 0x7f418419df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:37.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.037+0000 7f418beb2700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418419e540 con 0x7f4184100e20 2024-01-24T13:35:37.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.037+0000 7f418beb2700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418419e680 con 0x7f41840ffc30 2024-01-24T13:35:37.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.037+0000 7f418beb2700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418419e810 con 0x7f41841017c0 2024-01-24T13:35:37.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.037+0000 7f418944d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4184100e20 0x7f4184196f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:37.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.037+0000 7f418944d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4184100e20 0x7f4184196f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52410/0 (socket says 172.21.15.29:52410) 2024-01-24T13:35:37.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.037+0000 7f418944d700 1 -- 172.21.15.29:0/1003459297 learned_addr learned my addr 172.21.15.29:0/1003459297 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:37.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.037+0000 7f418a44f700 1 --2- 172.21.15.29:0/1003459297 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41841017c0 0x7f418419df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:37.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.037+0000 7f4189c4e700 1 --2- 172.21.15.29:0/1003459297 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41840ffc30 0x7f41841969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:37.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.037+0000 7f418944d700 1 -- 172.21.15.29:0/1003459297 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41841017c0 msgr2=0x7f418419df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:37.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.037+0000 7f418944d700 1 --2- 172.21.15.29:0/1003459297 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41841017c0 0x7f418419df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:37.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.037+0000 7f418944d700 1 -- 172.21.15.29:0/1003459297 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41840ffc30 msgr2=0x7f41841969c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:37.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.037+0000 7f418944d700 1 --2- 172.21.15.29:0/1003459297 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41840ffc30 0x7f41841969c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:37.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.037+0000 7f418944d700 1 -- 172.21.15.29:0/1003459297 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41740095f0 con 0x7f4184100e20 2024-01-24T13:35:37.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.037+0000 7f418944d700 1 --2- 172.21.15.29:0/1003459297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4184100e20 0x7f4184196f00 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4174009990 tx=0x7f4174004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:37.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.037+0000 7f417affd700 1 -- 172.21.15.29:0/1003459297 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417401d070 con 0x7f4184100e20 2024-01-24T13:35:37.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.037+0000 7f418beb2700 1 -- 172.21.15.29:0/1003459297 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f418419ea90 con 0x7f4184100e20 2024-01-24T13:35:37.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.037+0000 7f417affd700 1 -- 172.21.15.29:0/1003459297 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4174004090 con 0x7f4184100e20 2024-01-24T13:35:37.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.037+0000 7f417affd700 1 -- 172.21.15.29:0/1003459297 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4174004a00 con 0x7f4184100e20 2024-01-24T13:35:37.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.037+0000 7f418beb2700 1 -- 172.21.15.29:0/1003459297 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f418419ef40 con 0x7f4184100e20 2024-01-24T13:35:37.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.040+0000 7f417affd700 1 -- 172.21.15.29:0/1003459297 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f4174003bd0 con 0x7f4184100e20 2024-01-24T13:35:37.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.040+0000 7f417affd700 1 --2- 172.21.15.29:0/1003459297 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f41700a0520 0x7f41700a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:37.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.041+0000 7f4189c4e700 1 --2- 172.21.15.29:0/1003459297 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f41700a0520 0x7f41700a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:37.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.041+0000 7f417affd700 1 -- 172.21.15.29:0/1003459297 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f41740ccae0 con 0x7f4184100e20 2024-01-24T13:35:37.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.041+0000 7f4189c4e700 1 --2- 172.21.15.29:0/1003459297 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f41700a0520 0x7f41700a29e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4180005950 tx=0x7f41800058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:37.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.041+0000 7f4198046b80 1 Processor -- start 2024-01-24T13:35:37.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.042+0000 7f4198046b80 1 -- start start 2024-01-24T13:35:37.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.042+0000 7f4198046b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594c5d8a940 0x5594c5d8ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:37.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.042+0000 7f4198046b80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5594c5d8b2c0 0x5594c5d894d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:37.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.042+0000 7f4198046b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594c5d89ab0 0x5594c5d89f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:37.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.042+0000 7f4198046b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594c5cf8470 con 0x5594c5d8a940 2024-01-24T13:35:37.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.042+0000 7f4198046b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594c5cdca50 con 0x5594c5d89ab0 2024-01-24T13:35:37.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.042+0000 7f4198046b80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594c5cda410 con 0x5594c5d8b2c0 2024-01-24T13:35:37.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.042+0000 7f4189c4e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594c5d8a940 0x5594c5d8ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:37.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.042+0000 7f4189c4e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594c5d8a940 0x5594c5d8ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52412/0 (socket says 172.21.15.29:52412) 2024-01-24T13:35:37.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.042+0000 7f4189c4e700 1 -- 172.21.15.29:0/70087921 learned_addr learned my addr 172.21.15.29:0/70087921 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:37.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.042+0000 7f418944d700 1 --2- 172.21.15.29:0/70087921 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5594c5d8b2c0 0x5594c5d894d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:37.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.042+0000 7f418a44f700 1 --2- 172.21.15.29:0/70087921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594c5d89ab0 0x5594c5d89f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:37.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.042+0000 7f4189c4e700 1 -- 172.21.15.29:0/70087921 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5594c5d8b2c0 msgr2=0x5594c5d894d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T13:35:37.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.042+0000 7f4189c4e700 1 --2- 172.21.15.29:0/70087921 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5594c5d8b2c0 0x5594c5d894d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:37.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.042+0000 7f4189c4e700 1 -- 172.21.15.29:0/70087921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594c5d89ab0 msgr2=0x5594c5d89f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T13:35:37.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.042+0000 7f4189c4e700 1 --2- 172.21.15.29:0/70087921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594c5d89ab0 0x5594c5d89f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:37.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.042+0000 7f4189c4e700 1 -- 172.21.15.29:0/70087921 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5594c5b57360 con 0x5594c5d8a940 2024-01-24T13:35:37.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.042+0000 7f4189c4e700 1 --2- 172.21.15.29:0/70087921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594c5d8a940 0x5594c5d8ad60 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f418000b570 tx=0x7f4180012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38b1fc45f78b7bb0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:37.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.043+0000 7f416ffff700 1 -- 172.21.15.29:0/70087921 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4180011b50 con 0x5594c5d8a940 2024-01-24T13:35:37.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.043+0000 7f416ffff700 1 -- 172.21.15.29:0/70087921 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4180011cb0 con 0x5594c5d8a940 2024-01-24T13:35:37.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.043+0000 7f416ffff700 1 -- 172.21.15.29:0/70087921 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418001d610 con 0x5594c5d8a940 2024-01-24T13:35:37.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.043+0000 7f4198046b80 1 -- 172.21.15.29:0/70087921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594c5d8a940 msgr2=0x5594c5d8ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:37.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.043+0000 7f4198046b80 1 --2- 172.21.15.29:0/70087921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594c5d8a940 0x5594c5d8ad60 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f418000b570 tx=0x7f4180012000 comp rx=0 tx=0).stop 2024-01-24T13:35:37.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.043+0000 7f4198046b80 1 -- 172.21.15.29:0/70087921 shutdown_connections 2024-01-24T13:35:37.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.043+0000 7f4198046b80 1 --2- 172.21.15.29:0/70087921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594c5d89ab0 0x5594c5d89f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:37.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.043+0000 7f4198046b80 1 --2- 172.21.15.29:0/70087921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594c5d8a940 0x5594c5d8ad60 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:37.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.043+0000 7f4198046b80 1 --2- 172.21.15.29:0/70087921 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5594c5d8b2c0 0x5594c5d894d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:37.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.043+0000 7f4198046b80 1 -- 172.21.15.29:0/70087921 >> 172.21.15.29:0/70087921 conn(0x5594c5ce3d60 msgr2=0x5594c5d7ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:37.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.044+0000 7f4198046b80 1 -- 172.21.15.29:0/70087921 shutdown_connections 2024-01-24T13:35:37.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.044+0000 7f4198046b80 1 -- 172.21.15.29:0/70087921 wait complete. 2024-01-24T13:35:37.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.044+0000 7f4198046b80 1 -- start start 2024-01-24T13:35:37.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.044+0000 7f4198046b80 1 Processor -- start 2024-01-24T13:35:37.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.044+0000 7f4198046b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594c5d89ab0 0x5594c5d932d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:37.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.044+0000 7f4198046b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594c5d8a940 0x5594c5d93810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:37.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.045+0000 7f4198046b80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5594c5d8b2c0 0x5594c5d9a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:37.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.045+0000 7f4198046b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594c5cd5d00 con 0x5594c5d89ab0 2024-01-24T13:35:37.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.045+0000 7f4198046b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594c5c71d30 con 0x5594c5d8a940 2024-01-24T13:35:37.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.045+0000 7f4198046b80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594c5c719d0 con 0x5594c5d8b2c0 2024-01-24T13:35:37.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.045+0000 7f4198046b80 10 client.0 _ll_register_callbacks cb 0x5594c5c73240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-24T13:35:37.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.045+0000 7f4189c4e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594c5d89ab0 0x5594c5d932d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:37.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.045+0000 7f4189c4e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594c5d89ab0 0x5594c5d932d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52418/0 (socket says 172.21.15.29:52418) 2024-01-24T13:35:37.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.045+0000 7f4189c4e700 1 -- 172.21.15.29:0/657808274 learned_addr learned my addr 172.21.15.29:0/657808274 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:37.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.045+0000 7f418944d700 1 --2- 172.21.15.29:0/657808274 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594c5d8a940 0x5594c5d93810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:37.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.045+0000 7f418a44f700 1 --2- 172.21.15.29:0/657808274 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5594c5d8b2c0 0x5594c5d9a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:37.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.045+0000 7f4189c4e700 1 -- 172.21.15.29:0/657808274 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5594c5d8b2c0 msgr2=0x5594c5d9a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T13:35:37.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.045+0000 7f4189c4e700 1 --2- 172.21.15.29:0/657808274 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5594c5d8b2c0 0x5594c5d9a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:37.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.045+0000 7f4189c4e700 1 -- 172.21.15.29:0/657808274 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594c5d8a940 msgr2=0x5594c5d93810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T13:35:37.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.045+0000 7f4189c4e700 1 --2- 172.21.15.29:0/657808274 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594c5d8a940 0x5594c5d93810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:37.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.045+0000 7f4189c4e700 1 -- 172.21.15.29:0/657808274 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f418000fa10 con 0x5594c5d89ab0 2024-01-24T13:35:37.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.045+0000 7f4189c4e700 1 --2- 172.21.15.29:0/657808274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594c5d89ab0 0x5594c5d932d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f418001ed00 tx=0x7f4180011b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15f585e67c7e004e server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:37.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.045+0000 7f416d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-01-24T13:35:37.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.045+0000 7f416d7fa700 1 -- 172.21.15.29:0/657808274 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418002e070 con 0x5594c5d89ab0 2024-01-24T13:35:37.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.045+0000 7f416d7fa700 1 -- 172.21.15.29:0/657808274 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f418001e720 con 0x5594c5d89ab0 2024-01-24T13:35:37.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.045+0000 7f4198046b80 1 -- 172.21.15.29:0/657808274 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5594c5d78140 con 0x5594c5d89ab0 2024-01-24T13:35:37.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.046+0000 7f416d7fa700 1 -- 172.21.15.29:0/657808274 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f418001e8e0 con 0x5594c5d89ab0 2024-01-24T13:35:37.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.046+0000 7f4198046b80 10 client.14727 fetch_fsmap learned FSMap version 16 2024-01-24T13:35:37.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.046+0000 7f4198046b80 1 -- 172.21.15.29:0/657808274 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) v3 -- 0x5594c5b03b10 con 0x5594c5d89ab0 2024-01-24T13:35:37.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.046+0000 7f416d7fa700 1 -- 172.21.15.29:0/657808274 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5594c5d78140 con 0x5594c5d89ab0 2024-01-24T13:35:37.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.046+0000 7f416d7fa700 1 -- 172.21.15.29:0/657808274 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 16) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f4180004b50 con 0x5594c5d89ab0 2024-01-24T13:35:37.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.046+0000 7f4198046b80 10 client.14727 fetch_fsmap finished waiting for FSMap version 16 2024-01-24T13:35:37.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.046+0000 7f4198046b80 10 client.14727 resolve_mds: resolved name 'b' to daemon mds.b 2024-01-24T13:35:37.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.046+0000 7f4198046b80 20 client.14727 populate_metadata read hostname 'smithi029' 2024-01-24T13:35:37.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.046+0000 7f4198046b80 1 --2- 172.21.15.29:0/657808274 >> [v2:172.21.15.163:6838/2641650917,v1:172.21.15.163:6839/2641650917] conn(0x5594c5d9e480 0x5594c5d9d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:37.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.046+0000 7f4198046b80 4 client.14727 mds_command: new command op to 24548 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-24T13:35:37.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.046+0000 7f4198046b80 1 -- 172.21.15.29:0/657808274 --> [v2:172.21.15.163:6838/2641650917,v1:172.21.15.163:6839/2641650917] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5594c5cced80 con 0x5594c5d9e480 2024-01-24T13:35:37.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.046+0000 7f418944d700 1 --2- 172.21.15.29:0/657808274 >> [v2:172.21.15.163:6838/2641650917,v1:172.21.15.163:6839/2641650917] conn(0x5594c5d9e480 0x5594c5d9d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:37.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.047+0000 7f416d7fa700 10 client.14727 ms_handle_connect on v2:172.21.15.163:6838/2641650917 2024-01-24T13:35:37.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.047+0000 7f418944d700 1 --2- 172.21.15.29:0/657808274 >> [v2:172.21.15.163:6838/2641650917,v1:172.21.15.163:6839/2641650917] conn(0x5594c5d9e480 0x5594c5d9d890 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d607075d76e4a7d server_cookie=4ad1c6937942347b in_seq=0 out_seq=0 2024-01-24T13:35:37.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.048+0000 7f416d7fa700 1 -- 172.21.15.29:0/657808274 <== mds.0 v2:172.21.15.163:6838/2641650917 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x5594c5cced80 con 0x5594c5d9e480 2024-01-24T13:35:37.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.048+0000 7f416d7fa700 10 client.14727 handle_command_reply: tid=0 2024-01-24T13:35:37.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.048+0000 7f4198046b80 1 client.14727 shutdown 2024-01-24T13:35:37.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.048+0000 7f4198046b80 1 -- 172.21.15.29:0/657808274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594c5d89ab0 msgr2=0x5594c5d932d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:37.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.048+0000 7f4198046b80 1 --2- 172.21.15.29:0/657808274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594c5d89ab0 0x5594c5d932d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f418001ed00 tx=0x7f4180011b80 comp rx=0 tx=0).stop 2024-01-24T13:35:37.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.048+0000 7f4198046b80 1 -- 172.21.15.29:0/657808274 shutdown_connections 2024-01-24T13:35:37.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.048+0000 7f4198046b80 1 --2- 172.21.15.29:0/657808274 >> [v2:172.21.15.163:6838/2641650917,v1:172.21.15.163:6839/2641650917] conn(0x5594c5d9e480 0x5594c5d9d890 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:37.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.048+0000 7f4198046b80 1 --2- 172.21.15.29:0/657808274 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5594c5d8b2c0 0x5594c5d9a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:37.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.048+0000 7f4198046b80 1 --2- 172.21.15.29:0/657808274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594c5d89ab0 0x5594c5d932d0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:37.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.048+0000 7f4198046b80 1 --2- 172.21.15.29:0/657808274 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594c5d8a940 0x5594c5d93810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:37.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.048+0000 7f416d7fa700 0 client.14727 ms_handle_reset on v2:172.21.15.163:6838/2641650917 2024-01-24T13:35:37.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.048+0000 7f4198046b80 1 -- 172.21.15.29:0/657808274 >> 172.21.15.29:0/657808274 conn(0x5594c5ce3d60 msgr2=0x5594c5d7f610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:37.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.048+0000 7f4198046b80 1 -- 172.21.15.29:0/657808274 shutdown_connections 2024-01-24T13:35:37.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.048+0000 7f4198046b80 1 -- 172.21.15.29:0/657808274 wait complete. 2024-01-24T13:35:37.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.048+0000 7f4198046b80 20 client.14727 trim_cache size 0 max 16384 2024-01-24T13:35:37.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.067+0000 7f4198046b80 1 Processor -- start 2024-01-24T13:35:37.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.067+0000 7f4198046b80 1 -- start start 2024-01-24T13:35:37.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.067+0000 7f4198046b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594c5ce3d60 0x5594c5d91130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:37.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.067+0000 7f4198046b80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5594c5d89ab0 0x5594c5da71d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:37.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.067+0000 7f4198046b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594c5d8a940 0x5594c5d9d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:37.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.067+0000 7f4198046b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594c5b59650 con 0x5594c5d8a940 2024-01-24T13:35:37.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.067+0000 7f4198046b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594c5cced80 con 0x5594c5ce3d60 2024-01-24T13:35:37.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.067+0000 7f4198046b80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594c5c7f680 con 0x5594c5d89ab0 2024-01-24T13:35:37.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.067+0000 7f418a44f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594c5d8a940 0x5594c5d9d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:37.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.067+0000 7f418a44f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594c5d8a940 0x5594c5d9d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52434/0 (socket says 172.21.15.29:52434) 2024-01-24T13:35:37.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.067+0000 7f418a44f700 1 -- 172.21.15.29:0/659531013 learned_addr learned my addr 172.21.15.29:0/659531013 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:37.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.067+0000 7f4189c4e700 1 --2- 172.21.15.29:0/659531013 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594c5ce3d60 0x5594c5d91130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:37.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.067+0000 7f418944d700 1 --2- 172.21.15.29:0/659531013 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5594c5d89ab0 0x5594c5da71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:37.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.067+0000 7f418a44f700 1 -- 172.21.15.29:0/659531013 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5594c5d89ab0 msgr2=0x5594c5da71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T13:35:37.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.067+0000 7f418a44f700 1 --2- 172.21.15.29:0/659531013 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5594c5d89ab0 0x5594c5da71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:37.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.067+0000 7f418a44f700 1 -- 172.21.15.29:0/659531013 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594c5ce3d60 msgr2=0x5594c5d91130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T13:35:37.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.067+0000 7f418a44f700 1 --2- 172.21.15.29:0/659531013 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594c5ce3d60 0x5594c5d91130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:37.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.067+0000 7f418a44f700 1 -- 172.21.15.29:0/659531013 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5594c5c9be50 con 0x5594c5d8a940 2024-01-24T13:35:37.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.067+0000 7f4189c4e700 1 --2- 172.21.15.29:0/659531013 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594c5ce3d60 0x5594c5d91130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T13:35:37.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.067+0000 7f418a44f700 1 --2- 172.21.15.29:0/659531013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594c5d8a940 0x5594c5d9d890 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f417c013620 tx=0x7f417c007950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2da4b07c7813ae7 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:37.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.067+0000 7f4178ff9700 1 -- 172.21.15.29:0/659531013 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c013e50 con 0x5594c5d8a940 2024-01-24T13:35:37.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.068+0000 7f4178ff9700 1 -- 172.21.15.29:0/659531013 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f417c004bb0 con 0x5594c5d8a940 2024-01-24T13:35:37.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.068+0000 7f4178ff9700 1 -- 172.21.15.29:0/659531013 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c005190 con 0x5594c5d8a940 2024-01-24T13:35:37.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.068+0000 7f4198046b80 1 -- 172.21.15.29:0/659531013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594c5d8a940 msgr2=0x5594c5d9d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:37.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.068+0000 7f4198046b80 1 --2- 172.21.15.29:0/659531013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594c5d8a940 0x5594c5d9d890 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f417c013620 tx=0x7f417c007950 comp rx=0 tx=0).stop 2024-01-24T13:35:37.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.068+0000 7f4198046b80 1 -- 172.21.15.29:0/659531013 shutdown_connections 2024-01-24T13:35:37.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.068+0000 7f4198046b80 1 --2- 172.21.15.29:0/659531013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594c5d8a940 0x5594c5d9d890 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:37.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.068+0000 7f4198046b80 1 --2- 172.21.15.29:0/659531013 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594c5ce3d60 0x5594c5d91130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:37.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.068+0000 7f4198046b80 1 --2- 172.21.15.29:0/659531013 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5594c5d89ab0 0x5594c5da71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:37.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.068+0000 7f4198046b80 1 -- 172.21.15.29:0/659531013 >> 172.21.15.29:0/659531013 conn(0x5594c5d802a0 msgr2=0x5594c5d91fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:37.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.068+0000 7f4198046b80 1 -- 172.21.15.29:0/659531013 shutdown_connections 2024-01-24T13:35:37.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.068+0000 7f4198046b80 1 -- 172.21.15.29:0/659531013 wait complete. 2024-01-24T13:35:37.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.068+0000 7f4198046b80 1 -- start start 2024-01-24T13:35:37.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.069+0000 7f4198046b80 1 Processor -- start 2024-01-24T13:35:37.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.069+0000 7f4198046b80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5594c5ce3d60 0x5594c5da2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:37.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.069+0000 7f4198046b80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594c5d89ab0 0x5594c5da2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:37.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.069+0000 7f4198046b80 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594c5d8a940 0x5594c5da58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:37.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.069+0000 7f4198046b80 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594c5da5df0 con 0x5594c5d8a940 2024-01-24T13:35:37.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.069+0000 7f4198046b80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594c5da5f60 con 0x5594c5d89ab0 2024-01-24T13:35:37.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.069+0000 7f4198046b80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594c5da6120 con 0x5594c5ce3d60 2024-01-24T13:35:37.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.069+0000 7f4198046b80 10 client.0 _ll_register_callbacks cb 0x5594c5c73240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-24T13:35:37.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.069+0000 7f418a44f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594c5d8a940 0x5594c5da58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:37.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.069+0000 7f418a44f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594c5d8a940 0x5594c5da58b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52448/0 (socket says 172.21.15.29:52448) 2024-01-24T13:35:37.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.069+0000 7f418a44f700 1 -- 172.21.15.29:0/4288540182 learned_addr learned my addr 172.21.15.29:0/4288540182 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:37.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.069+0000 7f4189c4e700 1 --2- 172.21.15.29:0/4288540182 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5594c5ce3d60 0x5594c5da2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:37.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.070+0000 7f418944d700 1 --2- 172.21.15.29:0/4288540182 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594c5d89ab0 0x5594c5da2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:37.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.070+0000 7f418a44f700 1 -- 172.21.15.29:0/4288540182 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5594c5ce3d60 msgr2=0x5594c5da2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T13:35:37.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.070+0000 7f418a44f700 1 --2- 172.21.15.29:0/4288540182 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5594c5ce3d60 0x5594c5da2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:37.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.070+0000 7f418a44f700 1 -- 172.21.15.29:0/4288540182 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594c5d89ab0 msgr2=0x5594c5da2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T13:35:37.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.070+0000 7f418a44f700 1 --2- 172.21.15.29:0/4288540182 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594c5d89ab0 0x5594c5da2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:37.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.070+0000 7f418a44f700 1 -- 172.21.15.29:0/4288540182 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f417c0132d0 con 0x5594c5d8a940 2024-01-24T13:35:37.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.070+0000 7f418a44f700 1 --2- 172.21.15.29:0/4288540182 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594c5d8a940 0x5594c5da58b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f417c000f80 tx=0x7f417c004f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65fe4fbe9e56d09 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:37.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.070+0000 7f416d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-01-24T13:35:37.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.070+0000 7f416d7fa700 1 -- 172.21.15.29:0/4288540182 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c024070 con 0x5594c5d8a940 2024-01-24T13:35:37.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.070+0000 7f416d7fa700 1 -- 172.21.15.29:0/4288540182 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f417c016870 con 0x5594c5d8a940 2024-01-24T13:35:37.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.070+0000 7f4198046b80 1 -- 172.21.15.29:0/4288540182 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4180011cb0 con 0x5594c5d8a940 2024-01-24T13:35:37.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.070+0000 7f416d7fa700 1 -- 172.21.15.29:0/4288540182 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f417c013ed0 con 0x5594c5d8a940 2024-01-24T13:35:37.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.070+0000 7f4198046b80 10 client.14733 fetch_fsmap learned FSMap version 16 2024-01-24T13:35:37.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.070+0000 7f4198046b80 1 -- 172.21.15.29:0/4288540182 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) v3 -- 0x5594c5b5bc80 con 0x5594c5d8a940 2024-01-24T13:35:37.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.071+0000 7f416d7fa700 1 -- 172.21.15.29:0/4288540182 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f417c034020 con 0x5594c5d8a940 2024-01-24T13:35:37.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.071+0000 7f416d7fa700 1 -- 172.21.15.29:0/4288540182 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 16) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f417c029580 con 0x5594c5d8a940 2024-01-24T13:35:37.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.071+0000 7f4198046b80 10 client.14733 fetch_fsmap finished waiting for FSMap version 16 2024-01-24T13:35:37.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.071+0000 7f4198046b80 10 client.14733 resolve_mds: resolved name 'b' to daemon mds.b 2024-01-24T13:35:37.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.071+0000 7f4198046b80 20 client.14733 populate_metadata read hostname 'smithi029' 2024-01-24T13:35:37.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.071+0000 7f4198046b80 1 --2- 172.21.15.29:0/4288540182 >> [v2:172.21.15.163:6838/2641650917,v1:172.21.15.163:6839/2641650917] conn(0x5594c5d8b2c0 0x5594c5df1f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:37.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.071+0000 7f4198046b80 4 client.14733 mds_command: new command op to 24548 tid=0[{"prefix": "status"}] 2024-01-24T13:35:37.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.071+0000 7f4198046b80 1 -- 172.21.15.29:0/4288540182 --> [v2:172.21.15.163:6838/2641650917,v1:172.21.15.163:6839/2641650917] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5594c5c94f00 con 0x5594c5d8b2c0 2024-01-24T13:35:37.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.071+0000 7f4189c4e700 1 --2- 172.21.15.29:0/4288540182 >> [v2:172.21.15.163:6838/2641650917,v1:172.21.15.163:6839/2641650917] conn(0x5594c5d8b2c0 0x5594c5df1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:37.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.071+0000 7f416d7fa700 10 client.14733 ms_handle_connect on v2:172.21.15.163:6838/2641650917 2024-01-24T13:35:37.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.071+0000 7f4189c4e700 1 --2- 172.21.15.29:0/4288540182 >> [v2:172.21.15.163:6838/2641650917,v1:172.21.15.163:6839/2641650917] conn(0x5594c5d8b2c0 0x5594c5df1f60 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e81034c65fb45368 server_cookie=5aa4f8999ee80e18 in_seq=0 out_seq=0 2024-01-24T13:35:37.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.072+0000 7f416d7fa700 1 -- 172.21.15.29:0/4288540182 <== mds.0 v2:172.21.15.163:6838/2641650917 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x5594c5c94f00 con 0x5594c5d8b2c0 2024-01-24T13:35:37.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.072+0000 7f416d7fa700 10 client.14733 handle_command_reply: tid=0 2024-01-24T13:35:37.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.072+0000 7f4198046b80 1 client.14733 shutdown 2024-01-24T13:35:37.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.072+0000 7f4198046b80 1 -- 172.21.15.29:0/4288540182 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594c5d8a940 msgr2=0x5594c5da58b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:37.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.072+0000 7f4198046b80 1 --2- 172.21.15.29:0/4288540182 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594c5d8a940 0x5594c5da58b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f417c000f80 tx=0x7f417c004f50 comp rx=0 tx=0).stop 2024-01-24T13:35:37.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.072+0000 7f4198046b80 1 -- 172.21.15.29:0/4288540182 shutdown_connections 2024-01-24T13:35:37.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.072+0000 7f4198046b80 1 --2- 172.21.15.29:0/4288540182 >> [v2:172.21.15.163:6838/2641650917,v1:172.21.15.163:6839/2641650917] conn(0x5594c5d8b2c0 0x5594c5df1f60 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:37.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.072+0000 7f4198046b80 1 --2- 172.21.15.29:0/4288540182 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5594c5d8a940 0x5594c5da58b0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:37.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.072+0000 7f4198046b80 1 --2- 172.21.15.29:0/4288540182 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5594c5ce3d60 0x5594c5da2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:37.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.072+0000 7f4198046b80 1 --2- 172.21.15.29:0/4288540182 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594c5d89ab0 0x5594c5da2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:37.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.072+0000 7f416d7fa700 0 client.14733 ms_handle_reset on v2:172.21.15.163:6838/2641650917 2024-01-24T13:35:37.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.072+0000 7f4198046b80 1 -- 172.21.15.29:0/4288540182 >> 172.21.15.29:0/4288540182 conn(0x5594c5d802a0 msgr2=0x5594c5d89e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:37.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.072+0000 7f4198046b80 1 -- 172.21.15.29:0/4288540182 shutdown_connections 2024-01-24T13:35:37.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.072+0000 7f4198046b80 1 -- 172.21.15.29:0/4288540182 wait complete. 2024-01-24T13:35:37.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.072+0000 7f4198046b80 20 client.14733 trim_cache size 0 max 16384 2024-01-24T13:35:37.079 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-01-24T13:35:37.079 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "83afcf5e-babc-11ee-95b1-87774f69a715", 2024-01-24T13:35:37.079 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2024-01-24T13:35:37.079 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24548, 2024-01-24T13:35:37.079 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-01-24T13:35:37.080 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-01-24T13:35:37.080 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-01-24T13:35:37.080 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 6.6780513499999996, 2024-01-24T13:35:37.080 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 16, 2024-01-24T13:35:37.080 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 85, 2024-01-24T13:35:37.080 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 84, 2024-01-24T13:35:37.080 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 12.590952279 2024-01-24T13:35:37.080 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-01-24T13:35:37.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.072+0000 7f418beb2700 1 -- 172.21.15.29:0/1003459297 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f41700a0520 msgr2=0x7f41700a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:37.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.073+0000 7f418beb2700 1 --2- 172.21.15.29:0/1003459297 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f41700a0520 0x7f41700a29e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4180005950 tx=0x7f41800058e0 comp rx=0 tx=0).stop 2024-01-24T13:35:37.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.073+0000 7f418beb2700 1 -- 172.21.15.29:0/1003459297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4184100e20 msgr2=0x7f4184196f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:37.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.073+0000 7f418beb2700 1 --2- 172.21.15.29:0/1003459297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4184100e20 0x7f4184196f00 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4174009990 tx=0x7f4174004730 comp rx=0 tx=0).stop 2024-01-24T13:35:37.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.073+0000 7f418beb2700 1 -- 172.21.15.29:0/1003459297 shutdown_connections 2024-01-24T13:35:37.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.073+0000 7f418beb2700 1 --2- 172.21.15.29:0/1003459297 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41841017c0 0x7f418419df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:37.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.073+0000 7f418beb2700 1 --2- 172.21.15.29:0/1003459297 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41840ffc30 0x7f41841969c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:37.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.073+0000 7f418beb2700 1 --2- 172.21.15.29:0/1003459297 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f41700a0520 0x7f41700a29e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:37.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.073+0000 7f418beb2700 1 --2- 172.21.15.29:0/1003459297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4184100e20 0x7f4184196f00 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:37.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.073+0000 7f418beb2700 1 -- 172.21.15.29:0/1003459297 >> 172.21.15.29:0/1003459297 conn(0x7f41840fb1a0 msgr2=0x7f4184067e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:37.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.073+0000 7f418beb2700 1 -- 172.21.15.29:0/1003459297 shutdown_connections 2024-01-24T13:35:37.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:37.073+0000 7f418beb2700 1 -- 172.21.15.29:0/1003459297 wait complete. 2024-01-24T13:35:37.087 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-24T13:35:37.087 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-01-24T13:35:37.087 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 -- ceph fs ls 2024-01-24T13:35:37.331 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T13:35:37.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:37 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/1227071579' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T13:35:37.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:37 smithi163 ceph-mon[44969]: pgmap v228: 129 pgs: 118 active+clean, 11 unknown; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 7 op/s 2024-01-24T13:35:37.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:37 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/1791887298' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T13:35:37.793 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:37 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/1227071579' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T13:35:37.793 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:37 smithi029 ceph-mon[40651]: pgmap v228: 129 pgs: 118 active+clean, 11 unknown; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 7 op/s 2024-01-24T13:35:37.793 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:37 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/1791887298' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T13:35:37.804 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:37 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/1227071579' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T13:35:37.804 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:37 smithi187 ceph-mon[44911]: pgmap v228: 129 pgs: 118 active+clean, 11 unknown; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 7 op/s 2024-01-24T13:35:37.804 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:37 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/1791887298' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T13:35:38.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.350+0000 7f50bfae7700 1 -- 172.21.15.29:0/2954345087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50b810d9b0 msgr2=0x7f50b810fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:38.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.350+0000 7f50bfae7700 1 --2- 172.21.15.29:0/2954345087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50b810d9b0 0x7f50b810fda0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f50b000b0a0 tx=0x7f50b000b3b0 comp rx=0 tx=0).stop 2024-01-24T13:35:38.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.350+0000 7f50bfae7700 1 -- 172.21.15.29:0/2954345087 shutdown_connections 2024-01-24T13:35:38.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.350+0000 7f50bfae7700 1 --2- 172.21.15.29:0/2954345087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50b810d9b0 0x7f50b810fda0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:38.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.350+0000 7f50bfae7700 1 --2- 172.21.15.29:0/2954345087 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f50b8101c40 0x7f50b80ff230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:38.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.350+0000 7f50bfae7700 1 --2- 172.21.15.29:0/2954345087 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50b81020b0 0x7f50b80ff770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:38.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.350+0000 7f50bfae7700 1 -- 172.21.15.29:0/2954345087 >> 172.21.15.29:0/2954345087 conn(0x7f50b80761b0 msgr2=0x7f50b80fd560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:38.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.350+0000 7f50bfae7700 1 -- 172.21.15.29:0/2954345087 shutdown_connections 2024-01-24T13:35:38.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.350+0000 7f50bfae7700 1 -- 172.21.15.29:0/2954345087 wait complete. 2024-01-24T13:35:38.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.351+0000 7f50bfae7700 1 Processor -- start 2024-01-24T13:35:38.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.351+0000 7f50bfae7700 1 -- start start 2024-01-24T13:35:38.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.351+0000 7f50bfae7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50b8101c40 0x7f50b8199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:38.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.351+0000 7f50bfae7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50b81020b0 0x7f50b81996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:38.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.351+0000 7f50bfae7700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f50b810d9b0 0x7f50b819db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:38.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.351+0000 7f50bfae7700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b819e130 con 0x7f50b81020b0 2024-01-24T13:35:38.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.351+0000 7f50bfae7700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b819e2a0 con 0x7f50b810d9b0 2024-01-24T13:35:38.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.351+0000 7f50bfae7700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b819e460 con 0x7f50b8101c40 2024-01-24T13:35:38.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.351+0000 7f50bd082700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50b81020b0 0x7f50b81996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:38.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.351+0000 7f50bd082700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50b81020b0 0x7f50b81996b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52468/0 (socket says 172.21.15.29:52468) 2024-01-24T13:35:38.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.351+0000 7f50bd082700 1 -- 172.21.15.29:0/8307842 learned_addr learned my addr 172.21.15.29:0/8307842 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:38.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.351+0000 7f50bd883700 1 --2- 172.21.15.29:0/8307842 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50b8101c40 0x7f50b8199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:38.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.351+0000 7f50be084700 1 --2- 172.21.15.29:0/8307842 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f50b810d9b0 0x7f50b819db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:38.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.352+0000 7f50bd082700 1 -- 172.21.15.29:0/8307842 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50b8101c40 msgr2=0x7f50b8199170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:38.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.352+0000 7f50bd082700 1 --2- 172.21.15.29:0/8307842 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50b8101c40 0x7f50b8199170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:38.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.352+0000 7f50bd082700 1 -- 172.21.15.29:0/8307842 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f50b810d9b0 msgr2=0x7f50b819db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:38.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.352+0000 7f50bd082700 1 --2- 172.21.15.29:0/8307842 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f50b810d9b0 0x7f50b819db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:38.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.352+0000 7f50bd082700 1 -- 172.21.15.29:0/8307842 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b0009d00 con 0x7f50b81020b0 2024-01-24T13:35:38.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.352+0000 7f50bd082700 1 --2- 172.21.15.29:0/8307842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50b81020b0 0x7f50b81996b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f50a800e9a0 tx=0x7f50a800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:38.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.352+0000 7f50aeffd700 1 -- 172.21.15.29:0/8307842 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a8009790 con 0x7f50b81020b0 2024-01-24T13:35:38.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.352+0000 7f50aeffd700 1 -- 172.21.15.29:0/8307842 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f50a800cbd0 con 0x7f50b81020b0 2024-01-24T13:35:38.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.352+0000 7f50bfae7700 1 -- 172.21.15.29:0/8307842 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50b819e740 con 0x7f50b81020b0 2024-01-24T13:35:38.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.352+0000 7f50aeffd700 1 -- 172.21.15.29:0/8307842 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a8018680 con 0x7f50b81020b0 2024-01-24T13:35:38.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.352+0000 7f50bfae7700 1 -- 172.21.15.29:0/8307842 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50b81a5fa0 con 0x7f50b81020b0 2024-01-24T13:35:38.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.353+0000 7f50bfae7700 1 -- 172.21.15.29:0/8307842 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50b804ea40 con 0x7f50b81020b0 2024-01-24T13:35:38.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.355+0000 7f50aeffd700 1 -- 172.21.15.29:0/8307842 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f50a800cd40 con 0x7f50b81020b0 2024-01-24T13:35:38.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.355+0000 7f50aeffd700 1 --2- 172.21.15.29:0/8307842 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f50a40a0520 0x7f50a40a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:38.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.355+0000 7f50bd883700 1 --2- 172.21.15.29:0/8307842 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f50a40a0520 0x7f50a40a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:38.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.355+0000 7f50aeffd700 1 -- 172.21.15.29:0/8307842 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f50a8014070 con 0x7f50b81020b0 2024-01-24T13:35:38.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.356+0000 7f50bd883700 1 --2- 172.21.15.29:0/8307842 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f50a40a0520 0x7f50a40a29e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f50b4005950 tx=0x7f50b400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:38.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.359+0000 7f50aeffd700 1 -- 172.21.15.29:0/8307842 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f50a80959f0 con 0x7f50b81020b0 2024-01-24T13:35:38.621 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-01-24T13:35:38.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.619+0000 7f50bfae7700 1 -- 172.21.15.29:0/8307842 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f50b819a4d0 con 0x7f50b81020b0 2024-01-24T13:35:38.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.620+0000 7f50aeffd700 1 -- 172.21.15.29:0/8307842 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+88 (secure 0 0 0) 0x7f50a8099010 con 0x7f50b81020b0 2024-01-24T13:35:38.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.622+0000 7f50bfae7700 1 -- 172.21.15.29:0/8307842 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f50a40a0520 msgr2=0x7f50a40a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:38.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.622+0000 7f50bfae7700 1 --2- 172.21.15.29:0/8307842 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f50a40a0520 0x7f50a40a29e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f50b4005950 tx=0x7f50b400a300 comp rx=0 tx=0).stop 2024-01-24T13:35:38.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.622+0000 7f50bfae7700 1 -- 172.21.15.29:0/8307842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50b81020b0 msgr2=0x7f50b81996b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:38.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.622+0000 7f50bfae7700 1 --2- 172.21.15.29:0/8307842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50b81020b0 0x7f50b81996b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f50a800e9a0 tx=0x7f50a800ecb0 comp rx=0 tx=0).stop 2024-01-24T13:35:38.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.622+0000 7f50bfae7700 1 -- 172.21.15.29:0/8307842 shutdown_connections 2024-01-24T13:35:38.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.622+0000 7f50bfae7700 1 --2- 172.21.15.29:0/8307842 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f50b810d9b0 0x7f50b819db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:38.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.622+0000 7f50bfae7700 1 --2- 172.21.15.29:0/8307842 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50b8101c40 0x7f50b8199170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:38.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.622+0000 7f50bfae7700 1 --2- 172.21.15.29:0/8307842 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f50a40a0520 0x7f50a40a29e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:38.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.622+0000 7f50bfae7700 1 --2- 172.21.15.29:0/8307842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50b81020b0 0x7f50b81996b0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:38.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.622+0000 7f50bfae7700 1 -- 172.21.15.29:0/8307842 >> 172.21.15.29:0/8307842 conn(0x7f50b80761b0 msgr2=0x7f50b80fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:38.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.622+0000 7f50bfae7700 1 -- 172.21.15.29:0/8307842 shutdown_connections 2024-01-24T13:35:38.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:38.622+0000 7f50bfae7700 1 -- 172.21.15.29:0/8307842 wait complete. 2024-01-24T13:35:38.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:38 smithi163 ceph-mon[44969]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-01-24T13:35:38.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:38 smithi163 ceph-mon[44969]: Cluster is now healthy 2024-01-24T13:35:38.783 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:38 smithi029 ceph-mon[40651]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-01-24T13:35:38.784 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:38 smithi029 ceph-mon[40651]: Cluster is now healthy 2024-01-24T13:35:38.804 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:38 smithi187 ceph-mon[44911]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-01-24T13:35:38.804 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:38 smithi187 ceph-mon[44911]: Cluster is now healthy 2024-01-24T13:35:39.005 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-01-24T13:35:39.005 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-24T13:35:39.005 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2024-01-24T13:35:39.005 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-01-24T13:35:39.005 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-01-24T13:35:39.005 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-01-24T13:35:39.005 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-24T13:35:39.005 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-24T13:35:39.005 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-01-24T13:35:39.006 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T13:35:39.006 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2024-01-24T13:35:39.030 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-24T13:35:39.030 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-24T13:35:39.030 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2024-01-24T13:35:39.030 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2024-01-24T13:35:39.030 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2024-01-24T13:35:39.030 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2024-01-24T13:35:39.030 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T13:35:39.030 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2024-01-24T13:35:39.030 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2024-01-24T13:35:39.030 INFO:teuthology.orchestra.run.smithi029.stdout:3: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-24T13:35:39.030 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2024-01-24T13:35:39.030 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2024-01-24T13:35:39.030 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-24T13:35:39.031 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 41909sec preferred_lft 41909sec 2024-01-24T13:35:39.031 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link 2024-01-24T13:35:39.031 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2024-01-24T13:35:39.031 INFO:teuthology.orchestra.run.smithi029.stdout:4: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T13:35:39.031 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2024-01-24T13:35:39.031 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2024-01-24T13:35:39.031 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T13:35:39.031 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2024-01-24T13:35:39.031 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2024-01-24T13:35:39.031 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-01-24T13:35:39.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T13:35:39.032 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2024-01-24T13:35:39.032 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-01-24T13:35:39.032 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link add name ceph-brx type bridge 2024-01-24T13:35:39.032 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip addr flush dev ceph-brx 2024-01-24T13:35:39.032 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set ceph-brx up 2024-01-24T13:35:39.032 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-01-24T13:35:39.032 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-01-24T13:35:39.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T13:35:39.270 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-24T13:35:39.304 INFO:teuthology.orchestra.run.smithi029.stdout:1 2024-01-24T13:35:39.306 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T13:35:39.306 DEBUG:teuthology.orchestra.run.smithi029:> route 2024-01-24T13:35:39.388 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2024-01-24T13:35:39.388 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-24T13:35:39.388 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-24T13:35:39.388 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-24T13:35:39.389 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-24T13:35:39.389 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T13:35:39.389 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2024-01-24T13:35:39.389 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-01-24T13:35:39.389 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-24T13:35:39.389 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-24T13:35:39.390 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-24T13:35:39.390 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-01-24T13:35:39.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:39 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/8307842' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-01-24T13:35:39.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:39 smithi163 ceph-mon[44969]: pgmap v229: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 5 op/s 2024-01-24T13:35:39.800 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:39 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/8307842' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-01-24T13:35:39.800 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:39 smithi029 ceph-mon[40651]: pgmap v229: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 5 op/s 2024-01-24T13:35:39.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T13:35:39.802 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2024-01-24T13:35:39.804 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:39 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/8307842' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-01-24T13:35:39.804 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:39 smithi187 ceph-mon[44911]: pgmap v229: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 5 op/s 2024-01-24T13:35:39.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T13:35:39.826 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list-id 2024-01-24T13:35:39.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T13:35:39.889 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2024-01-24T13:35:39.890 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-01-24T13:35:39.890 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-01-24T13:35:39.890 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-01-24T13:35:39.890 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-01-24T13:35:40.003 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-01-24T13:35:40.003 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T13:35:40.004 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2024-01-24T13:35:40.004 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-01-24T13:35:40.004 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-01-24T13:35:40.004 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-01-24T13:35:40.004 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-01-24T13:35:40.004 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-01-24T13:35:40.004 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-01-24T13:35:40.004 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-01-24T13:35:40.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T13:35:40.241 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2024-01-24T13:35:40.241 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-01-24T13:35:40.241 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2024-01-24T13:35:40.241 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set dev brx.0 master ceph-brx 2024-01-24T13:35:40.241 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-01-24T13:35:40.355 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-01-24T13:35:40.355 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T13:35:40.355 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-01-24T13:35:40.375 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-01-24T13:35:40.375 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T13:35:40.375 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-01-24T13:35:40.436 DEBUG:teuthology.orchestra.run.smithi029:> sudo modprobe fuse 2024-01-24T13:35:40.512 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T13:35:40.512 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T13:35:40.588 INFO:teuthology.orchestra.run.smithi029.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T13:35:40.590 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T13:35:40.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T13:35:40.590 DEBUG:teuthology.orchestra.run.smithi029:> ls /sys/fs/fuse/connections 2024-01-24T13:35:40.653 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-24T13:35:40.654 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-01-24T13:35:40.696 DEBUG:teuthology.orchestra.run.smithi029:> sudo modprobe fuse 2024-01-24T13:35:40.732 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T13:35:40.732 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T13:35:40.804 INFO:teuthology.orchestra.run.smithi029.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T13:35:40.805 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi029.stderr:2024-01-24T13:35:40.804+0000 7ff035bd95c0 -1 init, newargv = 0x55aecbc50df0 newargc=15 2024-01-24T13:35:40.805 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi029.stderr:ceph-fuse[94890]: starting ceph client 2024-01-24T13:35:40.808 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T13:35:40.809 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T13:35:40.809 DEBUG:teuthology.orchestra.run.smithi029:> ls /sys/fs/fuse/connections 2024-01-24T13:35:40.812 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi029.stderr:ceph-fuse[94890]: starting fuse 2024-01-24T13:35:40.833 INFO:teuthology.orchestra.run.smithi029.stdout:124 2024-01-24T13:35:40.834 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [124] 2024-01-24T13:35:40.834 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-24T13:35:40.834 DEBUG:teuthology.orchestra.run.smithi029:> import glob 2024-01-24T13:35:40.834 DEBUG:teuthology.orchestra.run.smithi029:> import re 2024-01-24T13:35:40.834 DEBUG:teuthology.orchestra.run.smithi029:> import os 2024-01-24T13:35:40.834 DEBUG:teuthology.orchestra.run.smithi029:> import subprocess 2024-01-24T13:35:40.834 DEBUG:teuthology.orchestra.run.smithi029:> 2024-01-24T13:35:40.834 DEBUG:teuthology.orchestra.run.smithi029:> def _find_admin_socket(client_name): 2024-01-24T13:35:40.834 DEBUG:teuthology.orchestra.run.smithi029:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-01-24T13:35:40.834 DEBUG:teuthology.orchestra.run.smithi029:> files = glob.glob(asok_path) 2024-01-24T13:35:40.834 DEBUG:teuthology.orchestra.run.smithi029:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-01-24T13:35:40.835 DEBUG:teuthology.orchestra.run.smithi029:> 2024-01-24T13:35:40.835 DEBUG:teuthology.orchestra.run.smithi029:> # Given a non-glob path, it better be there 2024-01-24T13:35:40.835 DEBUG:teuthology.orchestra.run.smithi029:> if "*" not in asok_path: 2024-01-24T13:35:40.835 DEBUG:teuthology.orchestra.run.smithi029:> assert(len(files) == 1) 2024-01-24T13:35:40.835 DEBUG:teuthology.orchestra.run.smithi029:> return files[0] 2024-01-24T13:35:40.835 DEBUG:teuthology.orchestra.run.smithi029:> 2024-01-24T13:35:40.835 DEBUG:teuthology.orchestra.run.smithi029:> for f in files: 2024-01-24T13:35:40.835 DEBUG:teuthology.orchestra.run.smithi029:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-24T13:35:40.835 DEBUG:teuthology.orchestra.run.smithi029:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-24T13:35:40.835 DEBUG:teuthology.orchestra.run.smithi029:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-24T13:35:40.835 DEBUG:teuthology.orchestra.run.smithi029:> contents = proc_f.read() 2024-01-24T13:35:40.835 DEBUG:teuthology.orchestra.run.smithi029:> if mountpoint in contents: 2024-01-24T13:35:40.835 DEBUG:teuthology.orchestra.run.smithi029:> return f 2024-01-24T13:35:40.835 DEBUG:teuthology.orchestra.run.smithi029:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-24T13:35:40.835 DEBUG:teuthology.orchestra.run.smithi029:> 2024-01-24T13:35:40.836 DEBUG:teuthology.orchestra.run.smithi029:> print(_find_admin_socket("client.admin")) 2024-01-24T13:35:40.836 DEBUG:teuthology.orchestra.run.smithi029:> ' 2024-01-24T13:35:40.971 INFO:teuthology.orchestra.run.smithi029.stdout:/var/run/ceph/ceph-client.admin.94890.asok 2024-01-24T13:35:41.153 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.94890.asok 2024-01-24T13:35:41.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T13:35:41.153 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.94890.asok status 2024-01-24T13:35:41.333 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-01-24T13:35:41.333 INFO:teuthology.orchestra.run.smithi029.stdout: "metadata": { 2024-01-24T13:35:41.333 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_sha1": "4e3182a65800aed6d4aa2f6b86da3cfd72188134", 2024-01-24T13:35:41.333 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", 2024-01-24T13:35:41.333 INFO:teuthology.orchestra.run.smithi029.stdout: "entity_id": "admin", 2024-01-24T13:35:41.333 INFO:teuthology.orchestra.run.smithi029.stdout: "hostname": "smithi029", 2024-01-24T13:35:41.333 INFO:teuthology.orchestra.run.smithi029.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-01-24T13:35:41.333 INFO:teuthology.orchestra.run.smithi029.stdout: "pid": "94890", 2024-01-24T13:35:41.334 INFO:teuthology.orchestra.run.smithi029.stdout: "root": "/" 2024-01-24T13:35:41.334 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-01-24T13:35:41.334 INFO:teuthology.orchestra.run.smithi029.stdout: "dentry_count": 0, 2024-01-24T13:35:41.334 INFO:teuthology.orchestra.run.smithi029.stdout: "dentry_pinned_count": 0, 2024-01-24T13:35:41.334 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 14745, 2024-01-24T13:35:41.334 INFO:teuthology.orchestra.run.smithi029.stdout: "inst": { 2024-01-24T13:35:41.334 INFO:teuthology.orchestra.run.smithi029.stdout: "name": { 2024-01-24T13:35:41.334 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "client", 2024-01-24T13:35:41.334 INFO:teuthology.orchestra.run.smithi029.stdout: "num": 14745 2024-01-24T13:35:41.334 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-01-24T13:35:41.334 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": { 2024-01-24T13:35:41.334 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v1", 2024-01-24T13:35:41.334 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "192.168.0.1:0", 2024-01-24T13:35:41.334 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 3546025570 2024-01-24T13:35:41.335 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-01-24T13:35:41.335 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-01-24T13:35:41.335 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": { 2024-01-24T13:35:41.335 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v1", 2024-01-24T13:35:41.335 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "192.168.0.1:0", 2024-01-24T13:35:41.335 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 3546025570 2024-01-24T13:35:41.335 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-01-24T13:35:41.335 INFO:teuthology.orchestra.run.smithi029.stdout: "inst_str": "client.14745 192.168.0.1:0/3546025570", 2024-01-24T13:35:41.335 INFO:teuthology.orchestra.run.smithi029.stdout: "addr_str": "192.168.0.1:0/3546025570", 2024-01-24T13:35:41.335 INFO:teuthology.orchestra.run.smithi029.stdout: "inode_count": 1, 2024-01-24T13:35:41.335 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_epoch": 16, 2024-01-24T13:35:41.335 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_epoch": 85, 2024-01-24T13:35:41.335 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_epoch_barrier": 0, 2024-01-24T13:35:41.335 INFO:teuthology.orchestra.run.smithi029.stdout: "blocklisted": false, 2024-01-24T13:35:41.335 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs" 2024-01-24T13:35:41.336 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-01-24T13:35:41.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T13:35:41.344 DEBUG:teuthology.orchestra.run.smithi029:> stat --file-system '--printf=%T 2024-01-24T13:35:41.345 DEBUG:teuthology.orchestra.run.smithi029:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-01-24T13:35:41.364 INFO:teuthology.orchestra.run.smithi029.stdout:fuseblk 2024-01-24T13:35:41.364 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-01-24T13:35:41.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T13:35:41.365 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-01-24T13:35:41.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T13:35:41.439 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-01-24T13:35:41.561 INFO:teuthology.orchestra.run.smithi029.stdout:# file: . 2024-01-24T13:35:41.561 INFO:teuthology.orchestra.run.smithi029.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-01-24T13:35:41.562 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:35:41.562 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-24T13:35:41.562 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T13:35:41.585 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T13:35:41.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.772+0000 7f5780bca700 1 -- 172.21.15.29:0/1712288656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f577c100e10 msgr2=0x7f577c101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:41.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.772+0000 7f5780bca700 1 --2- 172.21.15.29:0/1712288656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f577c100e10 0x7f577c101270 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f57700099c0 tx=0x7f5770009cd0 comp rx=0 tx=0).stop 2024-01-24T13:35:41.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.772+0000 7f5780bca700 1 -- 172.21.15.29:0/1712288656 shutdown_connections 2024-01-24T13:35:41.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.772+0000 7f5780bca700 1 --2- 172.21.15.29:0/1712288656 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f577c1017b0 0x7f577c108060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:41.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.772+0000 7f5780bca700 1 --2- 172.21.15.29:0/1712288656 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f577c0ffc20 0x7f577c100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:41.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.772+0000 7f5780bca700 1 --2- 172.21.15.29:0/1712288656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f577c100e10 0x7f577c101270 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:41.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.772+0000 7f5780bca700 1 -- 172.21.15.29:0/1712288656 >> 172.21.15.29:0/1712288656 conn(0x7f577c0fb170 msgr2=0x7f577c0fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:41.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.772+0000 7f5780bca700 1 -- 172.21.15.29:0/1712288656 shutdown_connections 2024-01-24T13:35:41.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.772+0000 7f5780bca700 1 -- 172.21.15.29:0/1712288656 wait complete. 2024-01-24T13:35:41.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.773+0000 7f5780bca700 1 Processor -- start 2024-01-24T13:35:41.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.773+0000 7f5780bca700 1 -- start start 2024-01-24T13:35:41.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.773+0000 7f5780bca700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f577c0ffc20 0x7f577c068a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:41.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.773+0000 7f5780bca700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f577c100e10 0x7f577c19b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:41.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.773+0000 7f5780bca700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f577c1017b0 0x7f577c19dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:41.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.773+0000 7f5780bca700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f577c19e560 con 0x7f577c1017b0 2024-01-24T13:35:41.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.773+0000 7f5780bca700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f577c19e6d0 con 0x7f577c100e10 2024-01-24T13:35:41.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.773+0000 7f5780bca700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f577c19e890 con 0x7f577c0ffc20 2024-01-24T13:35:41.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.773+0000 7f577ad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f577c1017b0 0x7f577c19dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:41.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.773+0000 7f577ad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f577c1017b0 0x7f577c19dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44654/0 (socket says 172.21.15.29:44654) 2024-01-24T13:35:41.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.773+0000 7f577ad9d700 1 -- 172.21.15.29:0/3977515738 learned_addr learned my addr 172.21.15.29:0/3977515738 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:41.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.773+0000 7f577a59c700 1 --2- 172.21.15.29:0/3977515738 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f577c0ffc20 0x7f577c068a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:41.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.773+0000 7f5779d9b700 1 --2- 172.21.15.29:0/3977515738 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f577c100e10 0x7f577c19b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:41.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.774+0000 7f577ad9d700 1 -- 172.21.15.29:0/3977515738 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f577c0ffc20 msgr2=0x7f577c068a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:41.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.774+0000 7f577ad9d700 1 --2- 172.21.15.29:0/3977515738 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f577c0ffc20 0x7f577c068a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:41.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.774+0000 7f577ad9d700 1 -- 172.21.15.29:0/3977515738 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f577c100e10 msgr2=0x7f577c19b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:41.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.774+0000 7f577ad9d700 1 --2- 172.21.15.29:0/3977515738 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f577c100e10 0x7f577c19b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:41.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.774+0000 7f577ad9d700 1 -- 172.21.15.29:0/3977515738 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57700095f0 con 0x7f577c1017b0 2024-01-24T13:35:41.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.774+0000 7f577ad9d700 1 --2- 172.21.15.29:0/3977515738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f577c1017b0 0x7f577c19dd90 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5774007ee0 tx=0x7f577400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:41.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.774+0000 7f576b7fe700 1 -- 172.21.15.29:0/3977515738 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5774017070 con 0x7f577c1017b0 2024-01-24T13:35:41.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.774+0000 7f5780bca700 1 -- 172.21.15.29:0/3977515738 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f577c19eb70 con 0x7f577c1017b0 2024-01-24T13:35:41.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.774+0000 7f5780bca700 1 -- 172.21.15.29:0/3977515738 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f577c19f050 con 0x7f577c1017b0 2024-01-24T13:35:41.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.774+0000 7f576b7fe700 1 -- 172.21.15.29:0/3977515738 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f577400f040 con 0x7f577c1017b0 2024-01-24T13:35:41.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.774+0000 7f576b7fe700 1 -- 172.21.15.29:0/3977515738 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5774005530 con 0x7f577c1017b0 2024-01-24T13:35:41.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.775+0000 7f5780bca700 1 -- 172.21.15.29:0/3977515738 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f575c005320 con 0x7f577c1017b0 2024-01-24T13:35:41.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.777+0000 7f576b7fe700 1 -- 172.21.15.29:0/3977515738 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f577400db20 con 0x7f577c1017b0 2024-01-24T13:35:41.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.777+0000 7f576b7fe700 1 --2- 172.21.15.29:0/3977515738 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f57640a0570 0x7f57640a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:41.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.777+0000 7f577a59c700 1 --2- 172.21.15.29:0/3977515738 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f57640a0570 0x7f57640a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:41.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.777+0000 7f576b7fe700 1 -- 172.21.15.29:0/3977515738 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f57740cb1f0 con 0x7f577c1017b0 2024-01-24T13:35:41.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.778+0000 7f577a59c700 1 --2- 172.21.15.29:0/3977515738 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f57640a0570 0x7f57640a2a30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f577c100c70 tx=0x7f576c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:41.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.781+0000 7f576b7fe700 1 -- 172.21.15.29:0/3977515738 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f5774094c30 con 0x7f577c1017b0 2024-01-24T13:35:41.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.996+0000 7f5780bca700 1 -- 172.21.15.29:0/3977515738 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f575c005cc0 con 0x7f577c1017b0 2024-01-24T13:35:41.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:41.998+0000 7f576b7fe700 1 -- 172.21.15.29:0/3977515738 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23981 (secure 0 0 0) 0x7f5774098250 con 0x7f577c1017b0 2024-01-24T13:35:41.999 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:35:41.999 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":85,"fsid":"83afcf5e-babc-11ee-95b1-87774f69a715","created":"2024-01-24T13:29:33.175895+0000","modified":"2024-01-24T13:35:34.866792+0000","last_up_change":"2024-01-24T13:34:57.141902+0000","last_in_change":"2024-01-24T13:34:41.071943+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T13:32:21.264634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-24T13:35:28.761733+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-24T13:35:29.763293+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-01-24T13:35:32.858844+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9857a607-2e6f-41a5-a86a-795970be6210","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":2406629076},{"type":"v1","addr":"172.21.15.29:6803","nonce":2406629076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":2406629076},{"type":"v1","addr":"172.21.15.29:6805","nonce":2406629076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2406629076},{"type":"v1","addr":"172.21.15.29:6809","nonce":2406629076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":2406629076},{"type":"v1","addr":"172.21.15.29:6807","nonce":2406629076}]},"public_addr":"172.21.15.29:6803/2406629076","cluster_addr":"172.21.15.29:6805/2406629076","heartbeat_back_addr":"172.21.15.29:6809/2406629076","heartbeat_front_addr":"172.21.15.29:6807/2406629076","state":["exists","up"]},{"osd":1,"uuid":"8bb2ba48-23d4-4bbd-a912-ae74f9e7f6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":2217244566},{"type":"v1","addr":"172.21.15.163:6801","nonce":2217244566}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":2217244566},{"type":"v1","addr":"172.21.15.163:6803","nonce":2217244566}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":2217244566},{"type":"v1","addr":"172.21.15.163:6807","nonce":2217244566}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":2217244566},{"type":"v1","addr":"172.21.15.163:6805","nonce":2217244566}]},"public_addr":"172.21.15.163:6801/2217244566","cluster_addr":"172.21.15.163:6803/2217244566","heartbeat_back_addr":"172.21.15.163:6807/2217244566","heartbeat_front_addr":"172.21.15.163:6805/2217244566","state":["exists","up"]},{"osd":2,"uuid":"f0ed4131-cd29-4706-89ea-fde40f46ea83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3589694992},{"type":"v1","addr":"172.21.15.187:6801","nonce":3589694992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3589694992},{"type":"v1","addr":"172.21.15.187:6803","nonce":3589694992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3589694992},{"type":"v1","addr":"172.21.15.187:6807","nonce":3589694992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3589694992},{"type":"v1","addr":"172.21.15.187:6805","nonce":3589694992}]},"public_addr":"172.21.15.187:6801/3589694992","cluster_addr":"172.21.15.187:6803/3589694992","heartbeat_back_addr":"172.21.15.187:6807/3589694992","heartbeat_front_addr":"172.21.15.187:6805/3589694992","state":["exists","up"]},{"osd":3,"uuid":"da821310-b502-4b77-a5ce-0ffed0a3dafc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3417210415},{"type":"v1","addr":"172.21.15.29:6811","nonce":3417210415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":3417210415},{"type":"v1","addr":"172.21.15.29:6813","nonce":3417210415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3417210415},{"type":"v1","addr":"172.21.15.29:6817","nonce":3417210415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3417210415},{"type":"v1","addr":"172.21.15.29:6815","nonce":3417210415}]},"public_addr":"172.21.15.29:6811/3417210415","cluster_addr":"172.21.15.29:6813/3417210415","heartbeat_back_addr":"172.21.15.29:6817/3417210415","heartbeat_front_addr":"172.21.15.29:6815/3417210415","state":["exists","up"]},{"osd":4,"uuid":"8b3a348f-e7dd-4ccb-82a5-2d3f5d101da8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3283787183},{"type":"v1","addr":"172.21.15.163:6809","nonce":3283787183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":3283787183},{"type":"v1","addr":"172.21.15.163:6811","nonce":3283787183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3283787183},{"type":"v1","addr":"172.21.15.163:6815","nonce":3283787183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":3283787183},{"type":"v1","addr":"172.21.15.163:6813","nonce":3283787183}]},"public_addr":"172.21.15.163:6809/3283787183","cluster_addr":"172.21.15.163:6811/3283787183","heartbeat_back_addr":"172.21.15.163:6815/3283787183","heartbeat_front_addr":"172.21.15.163:6813/3283787183","state":["exists","up"]},{"osd":5,"uuid":"c26c7155-618b-4712-acdb-d4fd6c8b7f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":2259228817},{"type":"v1","addr":"172.21.15.187:6809","nonce":2259228817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":2259228817},{"type":"v1","addr":"172.21.15.187:6811","nonce":2259228817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":2259228817},{"type":"v1","addr":"172.21.15.187:6815","nonce":2259228817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":2259228817},{"type":"v1","addr":"172.21.15.187:6813","nonce":2259228817}]},"public_addr":"172.21.15.187:6809/2259228817","cluster_addr":"172.21.15.187:6811/2259228817","heartbeat_back_addr":"172.21.15.187:6815/2259228817","heartbeat_front_addr":"172.21.15.187:6813/2259228817","state":["exists","up"]},{"osd":6,"uuid":"042ac0b4-1fe1-444f-bf51-ad89613a2185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":938400701},{"type":"v1","addr":"172.21.15.29:6819","nonce":938400701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":938400701},{"type":"v1","addr":"172.21.15.29:6821","nonce":938400701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":938400701},{"type":"v1","addr":"172.21.15.29:6825","nonce":938400701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":938400701},{"type":"v1","addr":"172.21.15.29:6823","nonce":938400701}]},"public_addr":"172.21.15.29:6819/938400701","cluster_addr":"172.21.15.29:6821/938400701","heartbeat_back_addr":"172.21.15.29:6825/938400701","heartbeat_front_addr":"172.21.15.29:6823/938400701","state":["exists","up"]},{"osd":7,"uuid":"add3d09b-76f3-4a93-83a2-30ac94c3fad1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2783976092},{"type":"v1","addr":"172.21.15.163:6817","nonce":2783976092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2783976092},{"type":"v1","addr":"172.21.15.163:6819","nonce":2783976092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2783976092},{"type":"v1","addr":"172.21.15.163:6823","nonce":2783976092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2783976092},{"type":"v1","addr":"172.21.15.163:6821","nonce":2783976092}]},"public_addr":"172.21.15.163:6817/2783976092","cluster_addr":"172.21.15.163:6819/2783976092","heartbeat_back_addr":"172.21.15.163:6823/2783976092","heartbeat_front_addr":"172.21.15.163:6821/2783976092","state":["exists","up"]},{"osd":8,"uuid":"71044d6a-b55f-4bcf-94c5-79f71eb7b422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":2856803882},{"type":"v1","addr":"172.21.15.187:6817","nonce":2856803882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":2856803882},{"type":"v1","addr":"172.21.15.187:6819","nonce":2856803882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":2856803882},{"type":"v1","addr":"172.21.15.187:6823","nonce":2856803882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":2856803882},{"type":"v1","addr":"172.21.15.187:6821","nonce":2856803882}]},"public_addr":"172.21.15.187:6817/2856803882","cluster_addr":"172.21.15.187:6819/2856803882","heartbeat_back_addr":"172.21.15.187:6823/2856803882","heartbeat_front_addr":"172.21.15.187:6821/2856803882","state":["exists","up"]},{"osd":9,"uuid":"cf417a8e-e576-43de-b89c-f104ee3969ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1392813017},{"type":"v1","addr":"172.21.15.29:6827","nonce":1392813017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1392813017},{"type":"v1","addr":"172.21.15.29:6829","nonce":1392813017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1392813017},{"type":"v1","addr":"172.21.15.29:6833","nonce":1392813017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1392813017},{"type":"v1","addr":"172.21.15.29:6831","nonce":1392813017}]},"public_addr":"172.21.15.29:6827/1392813017","cluster_addr":"172.21.15.29:6829/1392813017","heartbeat_back_addr":"172.21.15.29:6833/1392813017","heartbeat_front_addr":"172.21.15.29:6831/1392813017","state":["exists","up"]},{"osd":10,"uuid":"bc7cfb9d-3bfa-45b2-929a-a8dc90f1a2a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":2523735721},{"type":"v1","addr":"172.21.15.163:6825","nonce":2523735721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":2523735721},{"type":"v1","addr":"172.21.15.163:6827","nonce":2523735721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":2523735721},{"type":"v1","addr":"172.21.15.163:6831","nonce":2523735721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":2523735721},{"type":"v1","addr":"172.21.15.163:6829","nonce":2523735721}]},"public_addr":"172.21.15.163:6825/2523735721","cluster_addr":"172.21.15.163:6827/2523735721","heartbeat_back_addr":"172.21.15.163:6831/2523735721","heartbeat_front_addr":"172.21.15.163:6829/2523735721","state":["exists","up"]},{"osd":11,"uuid":"1453684e-fe11-4cbc-a868-cf90b3a7a021","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1828122314},{"type":"v1","addr":"172.21.15.187:6825","nonce":1828122314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1828122314},{"type":"v1","addr":"172.21.15.187:6827","nonce":1828122314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":1828122314},{"type":"v1","addr":"172.21.15.187:6831","nonce":1828122314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":1828122314},{"type":"v1","addr":"172.21.15.187:6829","nonce":1828122314}]},"public_addr":"172.21.15.187:6825/1828122314","cluster_addr":"172.21.15.187:6827/1828122314","heartbeat_back_addr":"172.21.15.187:6831/1828122314","heartbeat_front_addr":"172.21.15.187:6829/1828122314","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:31:43.572543+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:32:00.759082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:32:17.705002+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:32:35.666788+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:32:53.221701+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:33:10.290988+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:33:27.946743+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:33:45.698074+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:34:02.750761+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:34:20.662278+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:34:38.233389+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:34:55.419540+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/1419474286":"2024-01-25T13:30:30.728638+0000","172.21.15.29:6800/835547951":"2024-01-25T13:30:08.775587+0000","172.21.15.29:6801/1996568496":"2024-01-25T13:30:30.728638+0000","172.21.15.29:0/2727527401":"2024-01-25T13:30:08.775587+0000","172.21.15.29:0/41509684":"2024-01-25T13:30:30.728638+0000","172.21.15.29:6800/1996568496":"2024-01-25T13:30:30.728638+0000","172.21.15.29:0/2833864599":"2024-01-25T13:30:30.728638+0000","172.21.15.29:0/168349482":"2024-01-25T13:30:08.775587+0000","172.21.15.29:6801/835547951":"2024-01-25T13:30:08.775587+0000","172.21.15.29:0/1142561996":"2024-01-25T13:30:08.775587+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T13:35:42.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.000+0000 7f5780bca700 1 -- 172.21.15.29:0/3977515738 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f57640a0570 msgr2=0x7f57640a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:42.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.000+0000 7f5780bca700 1 --2- 172.21.15.29:0/3977515738 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f57640a0570 0x7f57640a2a30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f577c100c70 tx=0x7f576c00b410 comp rx=0 tx=0).stop 2024-01-24T13:35:42.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.000+0000 7f5780bca700 1 -- 172.21.15.29:0/3977515738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f577c1017b0 msgr2=0x7f577c19dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:42.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.000+0000 7f5780bca700 1 --2- 172.21.15.29:0/3977515738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f577c1017b0 0x7f577c19dd90 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5774007ee0 tx=0x7f577400d3b0 comp rx=0 tx=0).stop 2024-01-24T13:35:42.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.000+0000 7f5780bca700 1 -- 172.21.15.29:0/3977515738 shutdown_connections 2024-01-24T13:35:42.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.000+0000 7f5780bca700 1 --2- 172.21.15.29:0/3977515738 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f57640a0570 0x7f57640a2a30 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:42.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.000+0000 7f5780bca700 1 --2- 172.21.15.29:0/3977515738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f577c1017b0 0x7f577c19dd90 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:42.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.000+0000 7f5780bca700 1 --2- 172.21.15.29:0/3977515738 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f577c0ffc20 0x7f577c068a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:42.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.000+0000 7f5780bca700 1 --2- 172.21.15.29:0/3977515738 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f577c100e10 0x7f577c19b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:42.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.000+0000 7f5780bca700 1 -- 172.21.15.29:0/3977515738 >> 172.21.15.29:0/3977515738 conn(0x7f577c0fb170 msgr2=0x7f577c104300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:42.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.000+0000 7f5780bca700 1 -- 172.21.15.29:0/3977515738 shutdown_connections 2024-01-24T13:35:42.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.000+0000 7f5780bca700 1 -- 172.21.15.29:0/3977515738 wait complete. 2024-01-24T13:35:42.013 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:41 smithi029 ceph-mon[40651]: pgmap v230: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 4 op/s 2024-01-24T13:35:42.015 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T13:32:21.264634+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T13:35:28.761733+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T13:35:29.763293+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-01-24T13:35:32.858844+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-24T13:35:42.015 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T13:35:42.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.154+0000 7fca06d5b700 1 -- 172.21.15.29:0/1098450655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca00100d50 msgr2=0x7fca00101170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:42.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.154+0000 7fca06d5b700 1 --2- 172.21.15.29:0/1098450655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca00100d50 0x7fca00101170 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc9f40099c0 tx=0x7fc9f4009cd0 comp rx=0 tx=0).stop 2024-01-24T13:35:42.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.154+0000 7fca06d5b700 1 -- 172.21.15.29:0/1098450655 shutdown_connections 2024-01-24T13:35:42.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.154+0000 7fca06d5b700 1 --2- 172.21.15.29:0/1098450655 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fca001028e0 0x7fca00109190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:42.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.154+0000 7fca06d5b700 1 --2- 172.21.15.29:0/1098450655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca00100d50 0x7fca00101170 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:42.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.154+0000 7fca06d5b700 1 --2- 172.21.15.29:0/1098450655 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca00101f40 0x7fca001023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:42.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.154+0000 7fca06d5b700 1 -- 172.21.15.29:0/1098450655 >> 172.21.15.29:0/1098450655 conn(0x7fca000fc2c0 msgr2=0x7fca000fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:42.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.154+0000 7fca06d5b700 1 -- 172.21.15.29:0/1098450655 shutdown_connections 2024-01-24T13:35:42.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.154+0000 7fca06d5b700 1 -- 172.21.15.29:0/1098450655 wait complete. 2024-01-24T13:35:42.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.155+0000 7fca06d5b700 1 Processor -- start 2024-01-24T13:35:42.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.155+0000 7fca06d5b700 1 -- start start 2024-01-24T13:35:42.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.155+0000 7fca06d5b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca00100d50 0x7fca00196a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:42.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.155+0000 7fca06d5b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fca00101f40 0x7fca00196fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:42.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.155+0000 7fca06d5b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca001028e0 0x7fca0019e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:42.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.155+0000 7fca06d5b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0019e610 con 0x7fca00100d50 2024-01-24T13:35:42.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.155+0000 7fca06d5b700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0019e750 con 0x7fca00101f40 2024-01-24T13:35:42.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.155+0000 7fca06d5b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0019e8e0 con 0x7fca001028e0 2024-01-24T13:35:42.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.155+0000 7fca04af7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca00100d50 0x7fca00196a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:42.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.155+0000 7fca04af7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca00100d50 0x7fca00196a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44668/0 (socket says 172.21.15.29:44668) 2024-01-24T13:35:42.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.155+0000 7fca04af7700 1 -- 172.21.15.29:0/3335017094 learned_addr learned my addr 172.21.15.29:0/3335017094 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:42.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.155+0000 7fc9fffff700 1 --2- 172.21.15.29:0/3335017094 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fca00101f40 0x7fca00196fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:42.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.155+0000 7fca052f8700 1 --2- 172.21.15.29:0/3335017094 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca001028e0 0x7fca0019e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:42.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.156+0000 7fca04af7700 1 -- 172.21.15.29:0/3335017094 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca001028e0 msgr2=0x7fca0019e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:42.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.156+0000 7fca04af7700 1 --2- 172.21.15.29:0/3335017094 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca001028e0 0x7fca0019e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:42.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.156+0000 7fca04af7700 1 -- 172.21.15.29:0/3335017094 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fca00101f40 msgr2=0x7fca00196fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:42.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.156+0000 7fca04af7700 1 --2- 172.21.15.29:0/3335017094 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fca00101f40 0x7fca00196fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:42.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.156+0000 7fca04af7700 1 -- 172.21.15.29:0/3335017094 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9f40095f0 con 0x7fca00100d50 2024-01-24T13:35:42.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.156+0000 7fca04af7700 1 --2- 172.21.15.29:0/3335017094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca00100d50 0x7fca00196a90 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4009990 tx=0x7fc9f4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:42.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.156+0000 7fc9fdffb700 1 -- 172.21.15.29:0/3335017094 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f401d070 con 0x7fca00100d50 2024-01-24T13:35:42.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.156+0000 7fc9fdffb700 1 -- 172.21.15.29:0/3335017094 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc9f4004010 con 0x7fca00100d50 2024-01-24T13:35:42.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.156+0000 7fca06d5b700 1 -- 172.21.15.29:0/3335017094 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca0019eb60 con 0x7fca00100d50 2024-01-24T13:35:42.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.156+0000 7fc9fdffb700 1 -- 172.21.15.29:0/3335017094 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f4022cd0 con 0x7fca00100d50 2024-01-24T13:35:42.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.157+0000 7fca06d5b700 1 -- 172.21.15.29:0/3335017094 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca0019f060 con 0x7fca00100d50 2024-01-24T13:35:42.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.158+0000 7fca06d5b700 1 -- 172.21.15.29:0/3335017094 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca00108330 con 0x7fca00100d50 2024-01-24T13:35:42.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.160+0000 7fc9fdffb700 1 -- 172.21.15.29:0/3335017094 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7fc9f4022470 con 0x7fca00100d50 2024-01-24T13:35:42.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.160+0000 7fc9fdffb700 1 --2- 172.21.15.29:0/3335017094 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fc9f00a05a0 0x7fc9f00a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:42.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.160+0000 7fc9fdffb700 1 -- 172.21.15.29:0/3335017094 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fc9f40cd160 con 0x7fca00100d50 2024-01-24T13:35:42.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.160+0000 7fc9fffff700 1 --2- 172.21.15.29:0/3335017094 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fc9f00a05a0 0x7fc9f00a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:42.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.161+0000 7fc9fffff700 1 --2- 172.21.15.29:0/3335017094 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fc9f00a05a0 0x7fc9f00a2a60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec005950 tx=0x7fc9ec00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:42.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.164+0000 7fc9fdffb700 1 -- 172.21.15.29:0/3335017094 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fc9f4096ba0 con 0x7fca00100d50 2024-01-24T13:35:42.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:41 smithi163 ceph-mon[44969]: pgmap v230: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 4 op/s 2024-01-24T13:35:42.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:41 smithi187 ceph-mon[44911]: pgmap v230: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 4 op/s 2024-01-24T13:35:42.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.376+0000 7fca06d5b700 1 -- 172.21.15.29:0/3335017094 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fca0002cee0 con 0x7fca00100d50 2024-01-24T13:35:42.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.376+0000 7fc9fdffb700 1 -- 172.21.15.29:0/3335017094 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc9f409a1c0 con 0x7fca00100d50 2024-01-24T13:35:42.377 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-01-24T13:35:42.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.379+0000 7fca06d5b700 1 -- 172.21.15.29:0/3335017094 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fc9f00a05a0 msgr2=0x7fc9f00a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:42.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.379+0000 7fca06d5b700 1 --2- 172.21.15.29:0/3335017094 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fc9f00a05a0 0x7fc9f00a2a60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec005950 tx=0x7fc9ec00b410 comp rx=0 tx=0).stop 2024-01-24T13:35:42.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.379+0000 7fca06d5b700 1 -- 172.21.15.29:0/3335017094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca00100d50 msgr2=0x7fca00196a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:42.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.379+0000 7fca06d5b700 1 --2- 172.21.15.29:0/3335017094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca00100d50 0x7fca00196a90 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4009990 tx=0x7fc9f4004730 comp rx=0 tx=0).stop 2024-01-24T13:35:42.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.379+0000 7fca06d5b700 1 -- 172.21.15.29:0/3335017094 shutdown_connections 2024-01-24T13:35:42.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.379+0000 7fca06d5b700 1 --2- 172.21.15.29:0/3335017094 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca001028e0 0x7fca0019e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:42.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.379+0000 7fca06d5b700 1 --2- 172.21.15.29:0/3335017094 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fc9f00a05a0 0x7fc9f00a2a60 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:42.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.379+0000 7fca06d5b700 1 --2- 172.21.15.29:0/3335017094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca00100d50 0x7fca00196a90 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:42.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.379+0000 7fca06d5b700 1 --2- 172.21.15.29:0/3335017094 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fca00101f40 0x7fca00196fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:42.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.379+0000 7fca06d5b700 1 -- 172.21.15.29:0/3335017094 >> 172.21.15.29:0/3335017094 conn(0x7fca000fc2c0 msgr2=0x7fca00105470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:42.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.379+0000 7fca06d5b700 1 -- 172.21.15.29:0/3335017094 shutdown_connections 2024-01-24T13:35:42.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.379+0000 7fca06d5b700 1 -- 172.21.15.29:0/3335017094 wait complete. 2024-01-24T13:35:42.394 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T13:35:42.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.537+0000 7fdf6e86a700 1 -- 172.21.15.29:0/3273612319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf68101810 msgr2=0x7fdf68103c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:42.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.537+0000 7fdf6e86a700 1 --2- 172.21.15.29:0/3273612319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf68101810 0x7fdf68103c00 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fdf58009990 tx=0x7fdf58009ca0 comp rx=0 tx=0).stop 2024-01-24T13:35:42.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.537+0000 7fdf6e86a700 1 -- 172.21.15.29:0/3273612319 shutdown_connections 2024-01-24T13:35:42.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.537+0000 7fdf6e86a700 1 --2- 172.21.15.29:0/3273612319 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdf68106a70 0x7fdf68108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:42.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.537+0000 7fdf6e86a700 1 --2- 172.21.15.29:0/3273612319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf68101810 0x7fdf68103c00 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:42.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.537+0000 7fdf6e86a700 1 --2- 172.21.15.29:0/3273612319 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdf68104140 0x7fdf68106530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:42.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.537+0000 7fdf6e86a700 1 -- 172.21.15.29:0/3273612319 >> 172.21.15.29:0/3273612319 conn(0x7fdf680fb1a0 msgr2=0x7fdf680fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:42.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.537+0000 7fdf6e86a700 1 -- 172.21.15.29:0/3273612319 shutdown_connections 2024-01-24T13:35:42.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.537+0000 7fdf6e86a700 1 -- 172.21.15.29:0/3273612319 wait complete. 2024-01-24T13:35:42.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.538+0000 7fdf6e86a700 1 Processor -- start 2024-01-24T13:35:42.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.538+0000 7fdf6e86a700 1 -- start start 2024-01-24T13:35:42.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.538+0000 7fdf6e86a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf68101810 0x7fdf68101180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:42.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.538+0000 7fdf6e86a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdf68104140 0x7fdf680ff7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:42.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.538+0000 7fdf6e86a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdf68106a70 0x7fdf680ffd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:42.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.538+0000 7fdf6e86a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf681004a0 con 0x7fdf68101810 2024-01-24T13:35:42.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.538+0000 7fdf6e86a700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf68100610 con 0x7fdf68106a70 2024-01-24T13:35:42.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.538+0000 7fdf6e86a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf681a0ad0 con 0x7fdf68104140 2024-01-24T13:35:42.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.538+0000 7fdf67fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf68101810 0x7fdf68101180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:42.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.538+0000 7fdf67fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf68101810 0x7fdf68101180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44692/0 (socket says 172.21.15.29:44692) 2024-01-24T13:35:42.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.538+0000 7fdf67fff700 1 -- 172.21.15.29:0/36944472 learned_addr learned my addr 172.21.15.29:0/36944472 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:42.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.538+0000 7fdf677fe700 1 --2- 172.21.15.29:0/36944472 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdf68104140 0x7fdf680ff7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:42.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.538+0000 7fdf6ce07700 1 --2- 172.21.15.29:0/36944472 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdf68106a70 0x7fdf680ffd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:42.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.539+0000 7fdf67fff700 1 -- 172.21.15.29:0/36944472 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdf68104140 msgr2=0x7fdf680ff7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:42.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.539+0000 7fdf67fff700 1 --2- 172.21.15.29:0/36944472 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdf68104140 0x7fdf680ff7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:42.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.539+0000 7fdf67fff700 1 -- 172.21.15.29:0/36944472 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdf68106a70 msgr2=0x7fdf680ffd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:42.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.539+0000 7fdf67fff700 1 --2- 172.21.15.29:0/36944472 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdf68106a70 0x7fdf680ffd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:42.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.539+0000 7fdf67fff700 1 -- 172.21.15.29:0/36944472 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf580095f0 con 0x7fdf68101810 2024-01-24T13:35:42.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.539+0000 7fdf6ce07700 1 --2- 172.21.15.29:0/36944472 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdf68106a70 0x7fdf680ffd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T13:35:42.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.539+0000 7fdf67fff700 1 --2- 172.21.15.29:0/36944472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf68101810 0x7fdf68101180 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fdf58005910 tx=0x7fdf5800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:42.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.539+0000 7fdf657fa700 1 -- 172.21.15.29:0/36944472 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf5801d070 con 0x7fdf68101810 2024-01-24T13:35:42.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.539+0000 7fdf657fa700 1 -- 172.21.15.29:0/36944472 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdf5800fb80 con 0x7fdf68101810 2024-01-24T13:35:42.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.539+0000 7fdf6e86a700 1 -- 172.21.15.29:0/36944472 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf681a0d50 con 0x7fdf68101810 2024-01-24T13:35:42.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.539+0000 7fdf657fa700 1 -- 172.21.15.29:0/36944472 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf58017650 con 0x7fdf68101810 2024-01-24T13:35:42.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.539+0000 7fdf6e86a700 1 -- 172.21.15.29:0/36944472 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf681a1170 con 0x7fdf68101810 2024-01-24T13:35:42.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.540+0000 7fdf6e86a700 1 -- 172.21.15.29:0/36944472 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf680fcd20 con 0x7fdf68101810 2024-01-24T13:35:42.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.542+0000 7fdf657fa700 1 -- 172.21.15.29:0/36944472 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7fdf58022470 con 0x7fdf68101810 2024-01-24T13:35:42.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.542+0000 7fdf657fa700 1 --2- 172.21.15.29:0/36944472 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fdf500a0520 0x7fdf500a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:42.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.542+0000 7fdf657fa700 1 -- 172.21.15.29:0/36944472 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fdf580cd510 con 0x7fdf68101810 2024-01-24T13:35:42.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.542+0000 7fdf677fe700 1 --2- 172.21.15.29:0/36944472 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fdf500a0520 0x7fdf500a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:42.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.543+0000 7fdf677fe700 1 --2- 172.21.15.29:0/36944472 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fdf500a0520 0x7fdf500a29e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c005950 tx=0x7fdf5c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:42.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.546+0000 7fdf657fa700 1 -- 172.21.15.29:0/36944472 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fdf58096f50 con 0x7fdf68101810 2024-01-24T13:35:42.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.758+0000 7fdf6e86a700 1 -- 172.21.15.29:0/36944472 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdf6802d090 con 0x7fdf68101810 2024-01-24T13:35:42.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.759+0000 7fdf657fa700 1 -- 172.21.15.29:0/36944472 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdf5809a570 con 0x7fdf68101810 2024-01-24T13:35:42.760 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-01-24T13:35:42.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.761+0000 7fdf6e86a700 1 -- 172.21.15.29:0/36944472 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fdf500a0520 msgr2=0x7fdf500a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:42.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.761+0000 7fdf6e86a700 1 --2- 172.21.15.29:0/36944472 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fdf500a0520 0x7fdf500a29e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c005950 tx=0x7fdf5c00b410 comp rx=0 tx=0).stop 2024-01-24T13:35:42.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.761+0000 7fdf6e86a700 1 -- 172.21.15.29:0/36944472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf68101810 msgr2=0x7fdf68101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:42.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.761+0000 7fdf6e86a700 1 --2- 172.21.15.29:0/36944472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf68101810 0x7fdf68101180 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fdf58005910 tx=0x7fdf5800f690 comp rx=0 tx=0).stop 2024-01-24T13:35:42.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.761+0000 7fdf6e86a700 1 -- 172.21.15.29:0/36944472 shutdown_connections 2024-01-24T13:35:42.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.761+0000 7fdf6e86a700 1 --2- 172.21.15.29:0/36944472 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdf68106a70 0x7fdf680ffd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:42.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.761+0000 7fdf6e86a700 1 --2- 172.21.15.29:0/36944472 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fdf500a0520 0x7fdf500a29e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:42.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.761+0000 7fdf6e86a700 1 --2- 172.21.15.29:0/36944472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf68101810 0x7fdf68101180 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:42.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.761+0000 7fdf6e86a700 1 --2- 172.21.15.29:0/36944472 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdf68104140 0x7fdf680ff7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:42.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.761+0000 7fdf6e86a700 1 -- 172.21.15.29:0/36944472 >> 172.21.15.29:0/36944472 conn(0x7fdf680fb1a0 msgr2=0x7fdf68102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:42.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.762+0000 7fdf6e86a700 1 -- 172.21.15.29:0/36944472 shutdown_connections 2024-01-24T13:35:42.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.762+0000 7fdf6e86a700 1 -- 172.21.15.29:0/36944472 wait complete. 2024-01-24T13:35:42.777 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T13:35:42.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.918+0000 7f60b735b700 1 -- 172.21.15.29:0/1523961661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60b0100e20 msgr2=0x7f60b0101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:42.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.918+0000 7f60b735b700 1 --2- 172.21.15.29:0/1523961661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60b0100e20 0x7f60b0101280 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f60a00099c0 tx=0x7f60a0009cd0 comp rx=0 tx=0).stop 2024-01-24T13:35:42.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.918+0000 7f60b735b700 1 -- 172.21.15.29:0/1523961661 shutdown_connections 2024-01-24T13:35:42.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.918+0000 7f60b735b700 1 --2- 172.21.15.29:0/1523961661 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60b01017c0 0x7f60b0108070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:42.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.918+0000 7f60b735b700 1 --2- 172.21.15.29:0/1523961661 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60b00ffc30 0x7f60b0100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:42.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.918+0000 7f60b735b700 1 --2- 172.21.15.29:0/1523961661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60b0100e20 0x7f60b0101280 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:42.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.918+0000 7f60b735b700 1 -- 172.21.15.29:0/1523961661 >> 172.21.15.29:0/1523961661 conn(0x7f60b00fb1a0 msgr2=0x7f60b00fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:42.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.918+0000 7f60b735b700 1 -- 172.21.15.29:0/1523961661 shutdown_connections 2024-01-24T13:35:42.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.918+0000 7f60b735b700 1 -- 172.21.15.29:0/1523961661 wait complete. 2024-01-24T13:35:42.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.919+0000 7f60b735b700 1 Processor -- start 2024-01-24T13:35:42.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.919+0000 7f60b735b700 1 -- start start 2024-01-24T13:35:42.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.919+0000 7f60b735b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60b00ffc30 0x7f60b0068a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:42.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.919+0000 7f60b735b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60b0100e20 0x7f60b019b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:42.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.919+0000 7f60b735b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60b01017c0 0x7f60b019ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:42.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.919+0000 7f60b735b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b019e550 con 0x7f60b0100e20 2024-01-24T13:35:42.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.919+0000 7f60b735b700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b019e6c0 con 0x7f60b00ffc30 2024-01-24T13:35:42.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.919+0000 7f60b735b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b019e880 con 0x7f60b01017c0 2024-01-24T13:35:42.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.919+0000 7f60b48f6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60b0100e20 0x7f60b019b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:42.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.919+0000 7f60b48f6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60b0100e20 0x7f60b019b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44704/0 (socket says 172.21.15.29:44704) 2024-01-24T13:35:42.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.919+0000 7f60b48f6700 1 -- 172.21.15.29:0/776470738 learned_addr learned my addr 172.21.15.29:0/776470738 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:42.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.919+0000 7f60b50f7700 1 --2- 172.21.15.29:0/776470738 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60b00ffc30 0x7f60b0068a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:42.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.919+0000 7f60b58f8700 1 --2- 172.21.15.29:0/776470738 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60b01017c0 0x7f60b019ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:42.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.919+0000 7f60b48f6700 1 -- 172.21.15.29:0/776470738 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60b01017c0 msgr2=0x7f60b019ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:42.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.919+0000 7f60b48f6700 1 --2- 172.21.15.29:0/776470738 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60b01017c0 0x7f60b019ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:42.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.919+0000 7f60b48f6700 1 -- 172.21.15.29:0/776470738 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60b00ffc30 msgr2=0x7f60b0068a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:42.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.919+0000 7f60b48f6700 1 --2- 172.21.15.29:0/776470738 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60b00ffc30 0x7f60b0068a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:42.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.919+0000 7f60b48f6700 1 -- 172.21.15.29:0/776470738 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a00095f0 con 0x7f60b0100e20 2024-01-24T13:35:42.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.919+0000 7f60b58f8700 1 --2- 172.21.15.29:0/776470738 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60b01017c0 0x7f60b019ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T13:35:42.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.920+0000 7f60b48f6700 1 --2- 172.21.15.29:0/776470738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60b0100e20 0x7f60b019b770 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f60a0009fd0 tx=0x7f60a000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:42.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.920+0000 7f60a67fc700 1 -- 172.21.15.29:0/776470738 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a001d070 con 0x7f60b0100e20 2024-01-24T13:35:42.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.920+0000 7f60b735b700 1 -- 172.21.15.29:0/776470738 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60b019eb00 con 0x7f60b0100e20 2024-01-24T13:35:42.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.920+0000 7f60a67fc700 1 -- 172.21.15.29:0/776470738 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f60a00054f0 con 0x7f60b0100e20 2024-01-24T13:35:42.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.920+0000 7f60a67fc700 1 -- 172.21.15.29:0/776470738 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a0004bd0 con 0x7f60b0100e20 2024-01-24T13:35:42.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.920+0000 7f60b735b700 1 -- 172.21.15.29:0/776470738 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b019ef20 con 0x7f60b0100e20 2024-01-24T13:35:42.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.921+0000 7f60b735b700 1 -- 172.21.15.29:0/776470738 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60b0106bb0 con 0x7f60b0100e20 2024-01-24T13:35:42.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.922+0000 7f60a67fc700 1 -- 172.21.15.29:0/776470738 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f60a000fb80 con 0x7f60b0100e20 2024-01-24T13:35:42.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.923+0000 7f60a67fc700 1 --2- 172.21.15.29:0/776470738 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f609c0a0520 0x7f609c0a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:42.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.923+0000 7f60a67fc700 1 -- 172.21.15.29:0/776470738 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f60a00cce00 con 0x7f60b0100e20 2024-01-24T13:35:42.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.923+0000 7f60b50f7700 1 --2- 172.21.15.29:0/776470738 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f609c0a0520 0x7f609c0a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:42.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.923+0000 7f60b50f7700 1 --2- 172.21.15.29:0/776470738 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f609c0a0520 0x7f609c0a29e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f60b0100c80 tx=0x7f60ac009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:42.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:42.926+0000 7f60a67fc700 1 -- 172.21.15.29:0/776470738 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f60a0096840 con 0x7f60b0100e20 2024-01-24T13:35:43.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:42 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/3977515738' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T13:35:43.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:42 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/3335017094' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-24T13:35:43.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:42 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/36944472' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-01-24T13:35:43.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.139+0000 7f60b735b700 1 -- 172.21.15.29:0/776470738 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f60b002ce00 con 0x7f60b0100e20 2024-01-24T13:35:43.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.140+0000 7f60a67fc700 1 -- 172.21.15.29:0/776470738 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) v1 ==== 100+0+11 (secure 0 0 0) 0x7f60a0026020 con 0x7f60b0100e20 2024-01-24T13:35:43.141 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2024-01-24T13:35:43.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.142+0000 7f60b735b700 1 -- 172.21.15.29:0/776470738 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f609c0a0520 msgr2=0x7f609c0a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:43.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.142+0000 7f60b735b700 1 --2- 172.21.15.29:0/776470738 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f609c0a0520 0x7f609c0a29e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f60b0100c80 tx=0x7f60ac009260 comp rx=0 tx=0).stop 2024-01-24T13:35:43.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.142+0000 7f60b735b700 1 -- 172.21.15.29:0/776470738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60b0100e20 msgr2=0x7f60b019b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:43.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.142+0000 7f60b735b700 1 --2- 172.21.15.29:0/776470738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60b0100e20 0x7f60b019b770 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f60a0009fd0 tx=0x7f60a000f690 comp rx=0 tx=0).stop 2024-01-24T13:35:43.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.142+0000 7f60b735b700 1 -- 172.21.15.29:0/776470738 shutdown_connections 2024-01-24T13:35:43.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.142+0000 7f60b735b700 1 --2- 172.21.15.29:0/776470738 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60b01017c0 0x7f60b019ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:43.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.142+0000 7f60b735b700 1 --2- 172.21.15.29:0/776470738 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60b00ffc30 0x7f60b0068a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:43.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.142+0000 7f60b735b700 1 --2- 172.21.15.29:0/776470738 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f609c0a0520 0x7f609c0a29e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:43.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.142+0000 7f60b735b700 1 --2- 172.21.15.29:0/776470738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60b0100e20 0x7f60b019b770 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:43.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.142+0000 7f60b735b700 1 -- 172.21.15.29:0/776470738 >> 172.21.15.29:0/776470738 conn(0x7f60b00fb1a0 msgr2=0x7f60b00fd480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:43.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.142+0000 7f60b735b700 1 -- 172.21.15.29:0/776470738 shutdown_connections 2024-01-24T13:35:43.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.142+0000 7f60b735b700 1 -- 172.21.15.29:0/776470738 wait complete. 2024-01-24T13:35:43.157 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-01-24T13:35:43.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:42 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/3977515738' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T13:35:43.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:42 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/3335017094' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-24T13:35:43.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:42 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/36944472' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-01-24T13:35:43.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.295+0000 7f13bed25700 1 -- 172.21.15.29:0/4066694491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13b81017c0 msgr2=0x7f13b8108070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:43.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.295+0000 7f13bed25700 1 --2- 172.21.15.29:0/4066694491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13b81017c0 0x7f13b8108070 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f13b000b0d0 tx=0x7f13b000b3e0 comp rx=0 tx=0).stop 2024-01-24T13:35:43.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.295+0000 7f13bed25700 1 -- 172.21.15.29:0/4066694491 shutdown_connections 2024-01-24T13:35:43.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.295+0000 7f13bed25700 1 --2- 172.21.15.29:0/4066694491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13b81017c0 0x7f13b8108070 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:43.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.295+0000 7f13bed25700 1 --2- 172.21.15.29:0/4066694491 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13b80ffc30 0x7f13b8100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:43.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.295+0000 7f13bed25700 1 --2- 172.21.15.29:0/4066694491 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f13b8100e20 0x7f13b8101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:43.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.295+0000 7f13bed25700 1 -- 172.21.15.29:0/4066694491 >> 172.21.15.29:0/4066694491 conn(0x7f13b80fb1a0 msgr2=0x7f13b80fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:43.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.295+0000 7f13bed25700 1 -- 172.21.15.29:0/4066694491 shutdown_connections 2024-01-24T13:35:43.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.296+0000 7f13bed25700 1 -- 172.21.15.29:0/4066694491 wait complete. 2024-01-24T13:35:43.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.296+0000 7f13bed25700 1 Processor -- start 2024-01-24T13:35:43.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.296+0000 7f13bed25700 1 -- start start 2024-01-24T13:35:43.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.296+0000 7f13bed25700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13b80ffc30 0x7f13b8196b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:43.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.296+0000 7f13bed25700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13b8100e20 0x7f13b8197060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:43.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.296+0000 7f13bed25700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f13b81017c0 0x7f13b819e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:43.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.296+0000 7f13bed25700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b819e6a0 con 0x7f13b80ffc30 2024-01-24T13:35:43.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.296+0000 7f13bed25700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b819e7e0 con 0x7f13b81017c0 2024-01-24T13:35:43.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.296+0000 7f13bed25700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b819e970 con 0x7f13b8100e20 2024-01-24T13:35:43.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.296+0000 7f13bcac1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13b80ffc30 0x7f13b8196b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:43.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.297+0000 7f13bcac1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13b80ffc30 0x7f13b8196b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44714/0 (socket says 172.21.15.29:44714) 2024-01-24T13:35:43.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.297+0000 7f13bcac1700 1 -- 172.21.15.29:0/1732464867 learned_addr learned my addr 172.21.15.29:0/1732464867 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:43.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.297+0000 7f13b7fff700 1 --2- 172.21.15.29:0/1732464867 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13b8100e20 0x7f13b8197060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:43.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.297+0000 7f13bd2c2700 1 --2- 172.21.15.29:0/1732464867 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f13b81017c0 0x7f13b819e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:43.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.297+0000 7f13bcac1700 1 -- 172.21.15.29:0/1732464867 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13b8100e20 msgr2=0x7f13b8197060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:43.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.297+0000 7f13bcac1700 1 --2- 172.21.15.29:0/1732464867 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13b8100e20 0x7f13b8197060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:43.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.297+0000 7f13bcac1700 1 -- 172.21.15.29:0/1732464867 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f13b81017c0 msgr2=0x7f13b819e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:43.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.297+0000 7f13bcac1700 1 --2- 172.21.15.29:0/1732464867 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f13b81017c0 0x7f13b819e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:43.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.297+0000 7f13bcac1700 1 -- 172.21.15.29:0/1732464867 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b0009d00 con 0x7f13b80ffc30 2024-01-24T13:35:43.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.297+0000 7f13bd2c2700 1 --2- 172.21.15.29:0/1732464867 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f13b81017c0 0x7f13b819e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T13:35:43.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.297+0000 7f13bcac1700 1 --2- 172.21.15.29:0/1732464867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13b80ffc30 0x7f13b8196b20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f13ac00e970 tx=0x7f13ac00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:43.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.297+0000 7f13b5ffb700 1 -- 172.21.15.29:0/1732464867 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ac009790 con 0x7f13b80ffc30 2024-01-24T13:35:43.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.297+0000 7f13b5ffb700 1 -- 172.21.15.29:0/1732464867 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f13ac00eec0 con 0x7f13b80ffc30 2024-01-24T13:35:43.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.297+0000 7f13bed25700 1 -- 172.21.15.29:0/1732464867 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13b819ec50 con 0x7f13b80ffc30 2024-01-24T13:35:43.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.297+0000 7f13b5ffb700 1 -- 172.21.15.29:0/1732464867 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ac003cf0 con 0x7f13b80ffc30 2024-01-24T13:35:43.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.298+0000 7f13bed25700 1 -- 172.21.15.29:0/1732464867 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13b819f160 con 0x7f13b80ffc30 2024-01-24T13:35:43.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.299+0000 7f13bed25700 1 -- 172.21.15.29:0/1732464867 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13b810b450 con 0x7f13b80ffc30 2024-01-24T13:35:43.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.300+0000 7f13b5ffb700 1 -- 172.21.15.29:0/1732464867 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f13ac00ccd0 con 0x7f13b80ffc30 2024-01-24T13:35:43.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.300+0000 7f13b5ffb700 1 --2- 172.21.15.29:0/1732464867 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f13a80a0520 0x7f13a80a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:43.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.301+0000 7f13b7fff700 1 --2- 172.21.15.29:0/1732464867 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f13a80a0520 0x7f13a80a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:43.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.301+0000 7f13b5ffb700 1 -- 172.21.15.29:0/1732464867 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f13ac014070 con 0x7f13b80ffc30 2024-01-24T13:35:43.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.301+0000 7f13b7fff700 1 --2- 172.21.15.29:0/1732464867 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f13a80a0520 0x7f13a80a29e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f13a4007be0 tx=0x7f13a40073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:43.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:42 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/3977515738' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T13:35:43.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:42 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/3335017094' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-24T13:35:43.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:42 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/36944472' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-01-24T13:35:43.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.305+0000 7f13b5ffb700 1 -- 172.21.15.29:0/1732464867 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f13ac0d1020 con 0x7f13b80ffc30 2024-01-24T13:35:43.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.516+0000 7f13bed25700 1 -- 172.21.15.29:0/1732464867 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f13b819f710 con 0x7f13b80ffc30 2024-01-24T13:35:43.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.517+0000 7f13b5ffb700 1 -- 172.21.15.29:0/1732464867 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v85) v1 ==== 103+0+11 (secure 0 0 0) 0x7f13ac0957c0 con 0x7f13b80ffc30 2024-01-24T13:35:43.518 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-01-24T13:35:43.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.519+0000 7f13bed25700 1 -- 172.21.15.29:0/1732464867 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f13a80a0520 msgr2=0x7f13a80a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:43.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.519+0000 7f13bed25700 1 --2- 172.21.15.29:0/1732464867 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f13a80a0520 0x7f13a80a29e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f13a4007be0 tx=0x7f13a40073d0 comp rx=0 tx=0).stop 2024-01-24T13:35:43.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.519+0000 7f13bed25700 1 -- 172.21.15.29:0/1732464867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13b80ffc30 msgr2=0x7f13b8196b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:43.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.519+0000 7f13bed25700 1 --2- 172.21.15.29:0/1732464867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13b80ffc30 0x7f13b8196b20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f13ac00e970 tx=0x7f13ac00ec80 comp rx=0 tx=0).stop 2024-01-24T13:35:43.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.519+0000 7f13bed25700 1 -- 172.21.15.29:0/1732464867 shutdown_connections 2024-01-24T13:35:43.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.519+0000 7f13bed25700 1 --2- 172.21.15.29:0/1732464867 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f13b81017c0 0x7f13b819e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:43.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.519+0000 7f13bed25700 1 --2- 172.21.15.29:0/1732464867 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f13a80a0520 0x7f13a80a29e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:43.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.519+0000 7f13bed25700 1 --2- 172.21.15.29:0/1732464867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13b80ffc30 0x7f13b8196b20 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:43.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.519+0000 7f13bed25700 1 --2- 172.21.15.29:0/1732464867 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13b8100e20 0x7f13b8197060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:43.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.519+0000 7f13bed25700 1 -- 172.21.15.29:0/1732464867 >> 172.21.15.29:0/1732464867 conn(0x7f13b80fb1a0 msgr2=0x7f13b8067ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:43.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.519+0000 7f13bed25700 1 -- 172.21.15.29:0/1732464867 shutdown_connections 2024-01-24T13:35:43.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.519+0000 7f13bed25700 1 -- 172.21.15.29:0/1732464867 wait complete. 2024-01-24T13:35:43.534 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T13:35:43.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.670+0000 7f6c82e0d700 1 -- 172.21.15.29:0/525094313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c7c101830 msgr2=0x7f6c7c1080e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:43.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.670+0000 7f6c82e0d700 1 --2- 172.21.15.29:0/525094313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c7c101830 0x7f6c7c1080e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6c740099c0 tx=0x7f6c74009cd0 comp rx=0 tx=0).stop 2024-01-24T13:35:43.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.671+0000 7f6c82e0d700 1 -- 172.21.15.29:0/525094313 shutdown_connections 2024-01-24T13:35:43.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.671+0000 7f6c82e0d700 1 --2- 172.21.15.29:0/525094313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c7c101830 0x7f6c7c1080e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:43.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.671+0000 7f6c82e0d700 1 --2- 172.21.15.29:0/525094313 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c7c0ffca0 0x7f6c7c1000c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:43.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.671+0000 7f6c82e0d700 1 --2- 172.21.15.29:0/525094313 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c7c100e90 0x7f6c7c1012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:43.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.671+0000 7f6c82e0d700 1 -- 172.21.15.29:0/525094313 >> 172.21.15.29:0/525094313 conn(0x7f6c7c0fb210 msgr2=0x7f6c7c0fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:43.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.671+0000 7f6c82e0d700 1 -- 172.21.15.29:0/525094313 shutdown_connections 2024-01-24T13:35:43.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.671+0000 7f6c82e0d700 1 -- 172.21.15.29:0/525094313 wait complete. 2024-01-24T13:35:43.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.671+0000 7f6c82e0d700 1 Processor -- start 2024-01-24T13:35:43.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.671+0000 7f6c82e0d700 1 -- start start 2024-01-24T13:35:43.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.671+0000 7f6c82e0d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c7c0ffca0 0x7f6c7c069c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:43.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.671+0000 7f6c82e0d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c7c100e90 0x7f6c7c19b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:43.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.672+0000 7f6c82e0d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c7c101830 0x7f6c7c19dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:43.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.672+0000 7f6c82e0d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c19e4b0 con 0x7f6c7c100e90 2024-01-24T13:35:43.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.672+0000 7f6c82e0d700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c19e620 con 0x7f6c7c0ffca0 2024-01-24T13:35:43.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.672+0000 7f6c82e0d700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c19e7e0 con 0x7f6c7c101830 2024-01-24T13:35:43.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.672+0000 7f6c7bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c7c100e90 0x7f6c7c19b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:43.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.672+0000 7f6c7bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c7c100e90 0x7f6c7c19b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44724/0 (socket says 172.21.15.29:44724) 2024-01-24T13:35:43.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.672+0000 7f6c7bfff700 1 -- 172.21.15.29:0/2770959707 learned_addr learned my addr 172.21.15.29:0/2770959707 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:43.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.672+0000 7f6c80ba9700 1 --2- 172.21.15.29:0/2770959707 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c7c0ffca0 0x7f6c7c069c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:43.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.672+0000 7f6c813aa700 1 --2- 172.21.15.29:0/2770959707 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c7c101830 0x7f6c7c19dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:43.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.672+0000 7f6c7bfff700 1 -- 172.21.15.29:0/2770959707 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c7c101830 msgr2=0x7f6c7c19dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:43.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.672+0000 7f6c7bfff700 1 --2- 172.21.15.29:0/2770959707 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c7c101830 0x7f6c7c19dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:43.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.672+0000 7f6c7bfff700 1 -- 172.21.15.29:0/2770959707 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c7c0ffca0 msgr2=0x7f6c7c069c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:43.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.672+0000 7f6c7bfff700 1 --2- 172.21.15.29:0/2770959707 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c7c0ffca0 0x7f6c7c069c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:43.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.672+0000 7f6c7bfff700 1 -- 172.21.15.29:0/2770959707 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c740095f0 con 0x7f6c7c100e90 2024-01-24T13:35:43.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.672+0000 7f6c813aa700 1 --2- 172.21.15.29:0/2770959707 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c7c101830 0x7f6c7c19dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T13:35:43.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.672+0000 7f6c7bfff700 1 --2- 172.21.15.29:0/2770959707 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c7c100e90 0x7f6c7c19b6d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6c6400d980 tx=0x7f6c6400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:43.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.672+0000 7f6c79ffb700 1 -- 172.21.15.29:0/2770959707 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c64004220 con 0x7f6c7c100e90 2024-01-24T13:35:43.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.672+0000 7f6c79ffb700 1 -- 172.21.15.29:0/2770959707 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6c64007640 con 0x7f6c7c100e90 2024-01-24T13:35:43.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.672+0000 7f6c82e0d700 1 -- 172.21.15.29:0/2770959707 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c7c19eac0 con 0x7f6c7c100e90 2024-01-24T13:35:43.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.672+0000 7f6c79ffb700 1 -- 172.21.15.29:0/2770959707 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c64005610 con 0x7f6c7c100e90 2024-01-24T13:35:43.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.673+0000 7f6c82e0d700 1 -- 172.21.15.29:0/2770959707 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c7c19f020 con 0x7f6c7c100e90 2024-01-24T13:35:43.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.674+0000 7f6c82e0d700 1 -- 172.21.15.29:0/2770959707 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c7c1071e0 con 0x7f6c7c100e90 2024-01-24T13:35:43.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.675+0000 7f6c79ffb700 1 -- 172.21.15.29:0/2770959707 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f6c64004750 con 0x7f6c7c100e90 2024-01-24T13:35:43.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.676+0000 7f6c79ffb700 1 --2- 172.21.15.29:0/2770959707 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f6c680a05a0 0x7f6c680a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:43.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.676+0000 7f6c79ffb700 1 -- 172.21.15.29:0/2770959707 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f6c640cb980 con 0x7f6c7c100e90 2024-01-24T13:35:43.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.676+0000 7f6c80ba9700 1 --2- 172.21.15.29:0/2770959707 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f6c680a05a0 0x7f6c680a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:43.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.676+0000 7f6c80ba9700 1 --2- 172.21.15.29:0/2770959707 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f6c680a05a0 0x7f6c680a2a60 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c009550 tx=0x7f6c6c0091b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:43.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.680+0000 7f6c79ffb700 1 -- 172.21.15.29:0/2770959707 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f6c64095330 con 0x7f6c7c100e90 2024-01-24T13:35:43.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.909+0000 7f6c82e0d700 1 -- 172.21.15.29:0/2770959707 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f6c7c02cc70 con 0x7f6c7c100e90 2024-01-24T13:35:43.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.909+0000 7f6c79ffb700 1 -- 172.21.15.29:0/2770959707 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v85) v1 ==== 81+0+587 (secure 0 0 0) 0x7f6c64016020 con 0x7f6c7c100e90 2024-01-24T13:35:43.910 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/1419474286 2024-01-25T13:30:30.728638+0000 2024-01-24T13:35:43.910 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6800/835547951 2024-01-25T13:30:08.775587+0000 2024-01-24T13:35:43.910 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6801/1996568496 2024-01-25T13:30:30.728638+0000 2024-01-24T13:35:43.910 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/2727527401 2024-01-25T13:30:08.775587+0000 2024-01-24T13:35:43.911 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/41509684 2024-01-25T13:30:30.728638+0000 2024-01-24T13:35:43.911 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6800/1996568496 2024-01-25T13:30:30.728638+0000 2024-01-24T13:35:43.911 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/2833864599 2024-01-25T13:30:30.728638+0000 2024-01-24T13:35:43.911 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/168349482 2024-01-25T13:30:08.775587+0000 2024-01-24T13:35:43.911 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6801/835547951 2024-01-25T13:30:08.775587+0000 2024-01-24T13:35:43.911 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/1142561996 2024-01-25T13:30:08.775587+0000 2024-01-24T13:35:43.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.912+0000 7f6c82e0d700 1 -- 172.21.15.29:0/2770959707 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f6c680a05a0 msgr2=0x7f6c680a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:43.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.912+0000 7f6c82e0d700 1 --2- 172.21.15.29:0/2770959707 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f6c680a05a0 0x7f6c680a2a60 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c009550 tx=0x7f6c6c0091b0 comp rx=0 tx=0).stop 2024-01-24T13:35:43.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.912+0000 7f6c82e0d700 1 -- 172.21.15.29:0/2770959707 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c7c100e90 msgr2=0x7f6c7c19b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:43.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.912+0000 7f6c82e0d700 1 --2- 172.21.15.29:0/2770959707 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c7c100e90 0x7f6c7c19b6d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6c6400d980 tx=0x7f6c6400dc90 comp rx=0 tx=0).stop 2024-01-24T13:35:43.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.912+0000 7f6c82e0d700 1 -- 172.21.15.29:0/2770959707 shutdown_connections 2024-01-24T13:35:43.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.912+0000 7f6c82e0d700 1 --2- 172.21.15.29:0/2770959707 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c7c101830 0x7f6c7c19dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:43.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.912+0000 7f6c82e0d700 1 --2- 172.21.15.29:0/2770959707 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c7c0ffca0 0x7f6c7c069c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:43.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.912+0000 7f6c82e0d700 1 --2- 172.21.15.29:0/2770959707 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f6c680a05a0 0x7f6c680a2a60 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:43.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.912+0000 7f6c82e0d700 1 --2- 172.21.15.29:0/2770959707 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c7c100e90 0x7f6c7c19b6d0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:43.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.912+0000 7f6c82e0d700 1 -- 172.21.15.29:0/2770959707 >> 172.21.15.29:0/2770959707 conn(0x7f6c7c0fb210 msgr2=0x7f6c7c104320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:43.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.912+0000 7f6c82e0d700 1 -- 172.21.15.29:0/2770959707 shutdown_connections 2024-01-24T13:35:43.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:43.912+0000 7f6c82e0d700 1 -- 172.21.15.29:0/2770959707 wait complete. 2024-01-24T13:35:43.914 INFO:teuthology.orchestra.run.smithi029.stderr:listed 10 entries 2024-01-24T13:35:43.926 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-24T13:35:43.926 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T13:35:43.946 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T13:35:44.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.126+0000 7f470d42b700 1 -- 172.21.15.29:0/321422713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47081017c0 msgr2=0x7f4708108070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:44.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.126+0000 7f470d42b700 1 --2- 172.21.15.29:0/321422713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47081017c0 0x7f4708108070 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f470000b0d0 tx=0x7f470000b3e0 comp rx=0 tx=0).stop 2024-01-24T13:35:44.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.126+0000 7f470d42b700 1 -- 172.21.15.29:0/321422713 shutdown_connections 2024-01-24T13:35:44.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.126+0000 7f470d42b700 1 --2- 172.21.15.29:0/321422713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47081017c0 0x7f4708108070 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:44.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.126+0000 7f470d42b700 1 --2- 172.21.15.29:0/321422713 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f47080ffc30 0x7f4708100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:44.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.126+0000 7f470d42b700 1 --2- 172.21.15.29:0/321422713 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4708100e20 0x7f4708101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:44.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.126+0000 7f470d42b700 1 -- 172.21.15.29:0/321422713 >> 172.21.15.29:0/321422713 conn(0x7f47080fb1a0 msgr2=0x7f47080fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:44.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.126+0000 7f470d42b700 1 -- 172.21.15.29:0/321422713 shutdown_connections 2024-01-24T13:35:44.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.126+0000 7f470d42b700 1 -- 172.21.15.29:0/321422713 wait complete. 2024-01-24T13:35:44.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.127+0000 7f470d42b700 1 Processor -- start 2024-01-24T13:35:44.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.127+0000 7f470d42b700 1 -- start start 2024-01-24T13:35:44.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.127+0000 7f470d42b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f47080ffc30 0x7f4708103b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:44.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.127+0000 7f470d42b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4708100e20 0x7f4708102200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:44.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.127+0000 7f470d42b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47081017c0 0x7f4708102740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:44.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.127+0000 7f470d42b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4708102ee0 con 0x7f47081017c0 2024-01-24T13:35:44.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.127+0000 7f470d42b700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4708103050 con 0x7f47080ffc30 2024-01-24T13:35:44.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.127+0000 7f470d42b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47081a0b20 con 0x7f4708100e20 2024-01-24T13:35:44.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.127+0000 7f47077fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47081017c0 0x7f4708102740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:44.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.127+0000 7f47077fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47081017c0 0x7f4708102740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44740/0 (socket says 172.21.15.29:44740) 2024-01-24T13:35:44.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.127+0000 7f47077fe700 1 -- 172.21.15.29:0/2278723833 learned_addr learned my addr 172.21.15.29:0/2278723833 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:44.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.127+0000 7f4706ffd700 1 --2- 172.21.15.29:0/2278723833 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f47080ffc30 0x7f4708103b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:44.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.127+0000 7f47067fc700 1 --2- 172.21.15.29:0/2278723833 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4708100e20 0x7f4708102200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:44.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.127+0000 7f47077fe700 1 -- 172.21.15.29:0/2278723833 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4708100e20 msgr2=0x7f4708102200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:44.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.127+0000 7f47077fe700 1 --2- 172.21.15.29:0/2278723833 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4708100e20 0x7f4708102200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:44.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.127+0000 7f47077fe700 1 -- 172.21.15.29:0/2278723833 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f47080ffc30 msgr2=0x7f4708103b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:44.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.127+0000 7f47077fe700 1 --2- 172.21.15.29:0/2278723833 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f47080ffc30 0x7f4708103b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:44.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.127+0000 7f47077fe700 1 -- 172.21.15.29:0/2278723833 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4700009d00 con 0x7f47081017c0 2024-01-24T13:35:44.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.128+0000 7f47077fe700 1 --2- 172.21.15.29:0/2278723833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47081017c0 0x7f4708102740 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4700008f10 tx=0x7f4700008f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:44.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.128+0000 7f46effff700 1 -- 172.21.15.29:0/2278723833 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470000e040 con 0x7f47081017c0 2024-01-24T13:35:44.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.128+0000 7f470d42b700 1 -- 172.21.15.29:0/2278723833 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47081a0d50 con 0x7f47081017c0 2024-01-24T13:35:44.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.128+0000 7f46effff700 1 -- 172.21.15.29:0/2278723833 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4700005070 con 0x7f47081017c0 2024-01-24T13:35:44.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.128+0000 7f46effff700 1 -- 172.21.15.29:0/2278723833 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470001b6a0 con 0x7f47081017c0 2024-01-24T13:35:44.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.128+0000 7f470d42b700 1 -- 172.21.15.29:0/2278723833 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47081a1170 con 0x7f47081017c0 2024-01-24T13:35:44.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.129+0000 7f470d42b700 1 -- 172.21.15.29:0/2278723833 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f470810b450 con 0x7f47081017c0 2024-01-24T13:35:44.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.130+0000 7f46effff700 1 -- 172.21.15.29:0/2278723833 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f47000051e0 con 0x7f47081017c0 2024-01-24T13:35:44.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.131+0000 7f46effff700 1 --2- 172.21.15.29:0/2278723833 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f46f00a0570 0x7f46f00a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:44.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.131+0000 7f4706ffd700 1 --2- 172.21.15.29:0/2278723833 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f46f00a0570 0x7f46f00a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:44.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.131+0000 7f46effff700 1 -- 172.21.15.29:0/2278723833 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f47000ccdf0 con 0x7f47081017c0 2024-01-24T13:35:44.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.131+0000 7f4706ffd700 1 --2- 172.21.15.29:0/2278723833 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f46f00a0570 0x7f46f00a2a30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4708100c80 tx=0x7f46f800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:44.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.135+0000 7f46effff700 1 -- 172.21.15.29:0/2278723833 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f4700096830 con 0x7f47081017c0 2024-01-24T13:35:44.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:43 smithi163 ceph-mon[44969]: pgmap v231: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 3 op/s 2024-01-24T13:35:44.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:43 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/776470738' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-01-24T13:35:44.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:43 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/1732464867' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-01-24T13:35:44.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:43 smithi187 ceph-mon[44911]: pgmap v231: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 3 op/s 2024-01-24T13:35:44.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:43 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/776470738' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-01-24T13:35:44.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:43 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/1732464867' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-01-24T13:35:44.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:43 smithi029 ceph-mon[40651]: pgmap v231: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 3 op/s 2024-01-24T13:35:44.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:43 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/776470738' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-01-24T13:35:44.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:43 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/1732464867' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-01-24T13:35:44.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.349+0000 7f470d42b700 1 -- 172.21.15.29:0/2278723833 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f470810b640 con 0x7f47081017c0 2024-01-24T13:35:44.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.350+0000 7f46effff700 1 -- 172.21.15.29:0/2278723833 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23981 (secure 0 0 0) 0x7f4700099e50 con 0x7f47081017c0 2024-01-24T13:35:44.351 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:35:44.351 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":85,"fsid":"83afcf5e-babc-11ee-95b1-87774f69a715","created":"2024-01-24T13:29:33.175895+0000","modified":"2024-01-24T13:35:34.866792+0000","last_up_change":"2024-01-24T13:34:57.141902+0000","last_in_change":"2024-01-24T13:34:41.071943+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T13:32:21.264634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-24T13:35:28.761733+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-24T13:35:29.763293+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-01-24T13:35:32.858844+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9857a607-2e6f-41a5-a86a-795970be6210","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":2406629076},{"type":"v1","addr":"172.21.15.29:6803","nonce":2406629076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":2406629076},{"type":"v1","addr":"172.21.15.29:6805","nonce":2406629076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2406629076},{"type":"v1","addr":"172.21.15.29:6809","nonce":2406629076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":2406629076},{"type":"v1","addr":"172.21.15.29:6807","nonce":2406629076}]},"public_addr":"172.21.15.29:6803/2406629076","cluster_addr":"172.21.15.29:6805/2406629076","heartbeat_back_addr":"172.21.15.29:6809/2406629076","heartbeat_front_addr":"172.21.15.29:6807/2406629076","state":["exists","up"]},{"osd":1,"uuid":"8bb2ba48-23d4-4bbd-a912-ae74f9e7f6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":2217244566},{"type":"v1","addr":"172.21.15.163:6801","nonce":2217244566}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":2217244566},{"type":"v1","addr":"172.21.15.163:6803","nonce":2217244566}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":2217244566},{"type":"v1","addr":"172.21.15.163:6807","nonce":2217244566}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":2217244566},{"type":"v1","addr":"172.21.15.163:6805","nonce":2217244566}]},"public_addr":"172.21.15.163:6801/2217244566","cluster_addr":"172.21.15.163:6803/2217244566","heartbeat_back_addr":"172.21.15.163:6807/2217244566","heartbeat_front_addr":"172.21.15.163:6805/2217244566","state":["exists","up"]},{"osd":2,"uuid":"f0ed4131-cd29-4706-89ea-fde40f46ea83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3589694992},{"type":"v1","addr":"172.21.15.187:6801","nonce":3589694992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3589694992},{"type":"v1","addr":"172.21.15.187:6803","nonce":3589694992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3589694992},{"type":"v1","addr":"172.21.15.187:6807","nonce":3589694992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3589694992},{"type":"v1","addr":"172.21.15.187:6805","nonce":3589694992}]},"public_addr":"172.21.15.187:6801/3589694992","cluster_addr":"172.21.15.187:6803/3589694992","heartbeat_back_addr":"172.21.15.187:6807/3589694992","heartbeat_front_addr":"172.21.15.187:6805/3589694992","state":["exists","up"]},{"osd":3,"uuid":"da821310-b502-4b77-a5ce-0ffed0a3dafc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3417210415},{"type":"v1","addr":"172.21.15.29:6811","nonce":3417210415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":3417210415},{"type":"v1","addr":"172.21.15.29:6813","nonce":3417210415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3417210415},{"type":"v1","addr":"172.21.15.29:6817","nonce":3417210415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3417210415},{"type":"v1","addr":"172.21.15.29:6815","nonce":3417210415}]},"public_addr":"172.21.15.29:6811/3417210415","cluster_addr":"172.21.15.29:6813/3417210415","heartbeat_back_addr":"172.21.15.29:6817/3417210415","heartbeat_front_addr":"172.21.15.29:6815/3417210415","state":["exists","up"]},{"osd":4,"uuid":"8b3a348f-e7dd-4ccb-82a5-2d3f5d101da8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3283787183},{"type":"v1","addr":"172.21.15.163:6809","nonce":3283787183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":3283787183},{"type":"v1","addr":"172.21.15.163:6811","nonce":3283787183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3283787183},{"type":"v1","addr":"172.21.15.163:6815","nonce":3283787183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":3283787183},{"type":"v1","addr":"172.21.15.163:6813","nonce":3283787183}]},"public_addr":"172.21.15.163:6809/3283787183","cluster_addr":"172.21.15.163:6811/3283787183","heartbeat_back_addr":"172.21.15.163:6815/3283787183","heartbeat_front_addr":"172.21.15.163:6813/3283787183","state":["exists","up"]},{"osd":5,"uuid":"c26c7155-618b-4712-acdb-d4fd6c8b7f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":2259228817},{"type":"v1","addr":"172.21.15.187:6809","nonce":2259228817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":2259228817},{"type":"v1","addr":"172.21.15.187:6811","nonce":2259228817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":2259228817},{"type":"v1","addr":"172.21.15.187:6815","nonce":2259228817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":2259228817},{"type":"v1","addr":"172.21.15.187:6813","nonce":2259228817}]},"public_addr":"172.21.15.187:6809/2259228817","cluster_addr":"172.21.15.187:6811/2259228817","heartbeat_back_addr":"172.21.15.187:6815/2259228817","heartbeat_front_addr":"172.21.15.187:6813/2259228817","state":["exists","up"]},{"osd":6,"uuid":"042ac0b4-1fe1-444f-bf51-ad89613a2185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":938400701},{"type":"v1","addr":"172.21.15.29:6819","nonce":938400701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":938400701},{"type":"v1","addr":"172.21.15.29:6821","nonce":938400701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":938400701},{"type":"v1","addr":"172.21.15.29:6825","nonce":938400701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":938400701},{"type":"v1","addr":"172.21.15.29:6823","nonce":938400701}]},"public_addr":"172.21.15.29:6819/938400701","cluster_addr":"172.21.15.29:6821/938400701","heartbeat_back_addr":"172.21.15.29:6825/938400701","heartbeat_front_addr":"172.21.15.29:6823/938400701","state":["exists","up"]},{"osd":7,"uuid":"add3d09b-76f3-4a93-83a2-30ac94c3fad1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2783976092},{"type":"v1","addr":"172.21.15.163:6817","nonce":2783976092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2783976092},{"type":"v1","addr":"172.21.15.163:6819","nonce":2783976092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2783976092},{"type":"v1","addr":"172.21.15.163:6823","nonce":2783976092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2783976092},{"type":"v1","addr":"172.21.15.163:6821","nonce":2783976092}]},"public_addr":"172.21.15.163:6817/2783976092","cluster_addr":"172.21.15.163:6819/2783976092","heartbeat_back_addr":"172.21.15.163:6823/2783976092","heartbeat_front_addr":"172.21.15.163:6821/2783976092","state":["exists","up"]},{"osd":8,"uuid":"71044d6a-b55f-4bcf-94c5-79f71eb7b422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":2856803882},{"type":"v1","addr":"172.21.15.187:6817","nonce":2856803882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":2856803882},{"type":"v1","addr":"172.21.15.187:6819","nonce":2856803882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":2856803882},{"type":"v1","addr":"172.21.15.187:6823","nonce":2856803882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":2856803882},{"type":"v1","addr":"172.21.15.187:6821","nonce":2856803882}]},"public_addr":"172.21.15.187:6817/2856803882","cluster_addr":"172.21.15.187:6819/2856803882","heartbeat_back_addr":"172.21.15.187:6823/2856803882","heartbeat_front_addr":"172.21.15.187:6821/2856803882","state":["exists","up"]},{"osd":9,"uuid":"cf417a8e-e576-43de-b89c-f104ee3969ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1392813017},{"type":"v1","addr":"172.21.15.29:6827","nonce":1392813017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1392813017},{"type":"v1","addr":"172.21.15.29:6829","nonce":1392813017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1392813017},{"type":"v1","addr":"172.21.15.29:6833","nonce":1392813017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1392813017},{"type":"v1","addr":"172.21.15.29:6831","nonce":1392813017}]},"public_addr":"172.21.15.29:6827/1392813017","cluster_addr":"172.21.15.29:6829/1392813017","heartbeat_back_addr":"172.21.15.29:6833/1392813017","heartbeat_front_addr":"172.21.15.29:6831/1392813017","state":["exists","up"]},{"osd":10,"uuid":"bc7cfb9d-3bfa-45b2-929a-a8dc90f1a2a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":2523735721},{"type":"v1","addr":"172.21.15.163:6825","nonce":2523735721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":2523735721},{"type":"v1","addr":"172.21.15.163:6827","nonce":2523735721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":2523735721},{"type":"v1","addr":"172.21.15.163:6831","nonce":2523735721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":2523735721},{"type":"v1","addr":"172.21.15.163:6829","nonce":2523735721}]},"public_addr":"172.21.15.163:6825/2523735721","cluster_addr":"172.21.15.163:6827/2523735721","heartbeat_back_addr":"172.21.15.163:6831/2523735721","heartbeat_front_addr":"172.21.15.163:6829/2523735721","state":["exists","up"]},{"osd":11,"uuid":"1453684e-fe11-4cbc-a868-cf90b3a7a021","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1828122314},{"type":"v1","addr":"172.21.15.187:6825","nonce":1828122314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1828122314},{"type":"v1","addr":"172.21.15.187:6827","nonce":1828122314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":1828122314},{"type":"v1","addr":"172.21.15.187:6831","nonce":1828122314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":1828122314},{"type":"v1","addr":"172.21.15.187:6829","nonce":1828122314}]},"public_addr":"172.21.15.187:6825/1828122314","cluster_addr":"172.21.15.187:6827/1828122314","heartbeat_back_addr":"172.21.15.187:6831/1828122314","heartbeat_front_addr":"172.21.15.187:6829/1828122314","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:31:43.572543+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:32:00.759082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:32:17.705002+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:32:35.666788+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:32:53.221701+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:33:10.290988+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:33:27.946743+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:33:45.698074+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:34:02.750761+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:34:20.662278+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:34:38.233389+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:34:55.419540+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/1419474286":"2024-01-25T13:30:30.728638+0000","172.21.15.29:6800/835547951":"2024-01-25T13:30:08.775587+0000","172.21.15.29:6801/1996568496":"2024-01-25T13:30:30.728638+0000","172.21.15.29:0/2727527401":"2024-01-25T13:30:08.775587+0000","172.21.15.29:0/41509684":"2024-01-25T13:30:30.728638+0000","172.21.15.29:6800/1996568496":"2024-01-25T13:30:30.728638+0000","172.21.15.29:0/2833864599":"2024-01-25T13:30:30.728638+0000","172.21.15.29:0/168349482":"2024-01-25T13:30:08.775587+0000","172.21.15.29:6801/835547951":"2024-01-25T13:30:08.775587+0000","172.21.15.29:0/1142561996":"2024-01-25T13:30:08.775587+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T13:35:44.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.352+0000 7f470d42b700 1 -- 172.21.15.29:0/2278723833 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f46f00a0570 msgr2=0x7f46f00a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:44.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.352+0000 7f470d42b700 1 --2- 172.21.15.29:0/2278723833 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f46f00a0570 0x7f46f00a2a30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4708100c80 tx=0x7f46f800b410 comp rx=0 tx=0).stop 2024-01-24T13:35:44.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.352+0000 7f470d42b700 1 -- 172.21.15.29:0/2278723833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47081017c0 msgr2=0x7f4708102740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:44.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.352+0000 7f470d42b700 1 --2- 172.21.15.29:0/2278723833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47081017c0 0x7f4708102740 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4700008f10 tx=0x7f4700008f40 comp rx=0 tx=0).stop 2024-01-24T13:35:44.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.352+0000 7f470d42b700 1 -- 172.21.15.29:0/2278723833 shutdown_connections 2024-01-24T13:35:44.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.352+0000 7f470d42b700 1 --2- 172.21.15.29:0/2278723833 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f46f00a0570 0x7f46f00a2a30 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:44.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.352+0000 7f470d42b700 1 --2- 172.21.15.29:0/2278723833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47081017c0 0x7f4708102740 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:44.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.352+0000 7f470d42b700 1 --2- 172.21.15.29:0/2278723833 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f47080ffc30 0x7f4708103b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:44.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.352+0000 7f470d42b700 1 --2- 172.21.15.29:0/2278723833 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4708100e20 0x7f4708102200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:44.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.352+0000 7f470d42b700 1 -- 172.21.15.29:0/2278723833 >> 172.21.15.29:0/2278723833 conn(0x7f47080fb1a0 msgr2=0x7f4708069180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:44.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.353+0000 7f470d42b700 1 -- 172.21.15.29:0/2278723833 shutdown_connections 2024-01-24T13:35:44.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.353+0000 7f470d42b700 1 -- 172.21.15.29:0/2278723833 wait complete. 2024-01-24T13:35:44.367 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T13:32:21.264634+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T13:35:28.761733+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T13:35:29.763293+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-01-24T13:35:32.858844+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-24T13:35:44.367 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T13:35:44.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.505+0000 7f52f3878700 1 -- 172.21.15.29:0/1782253200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52ec104160 msgr2=0x7f52ec106550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:44.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.505+0000 7f52f3878700 1 --2- 172.21.15.29:0/1782253200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52ec104160 0x7f52ec106550 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f52dc009990 tx=0x7f52dc009ca0 comp rx=0 tx=0).stop 2024-01-24T13:35:44.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.505+0000 7f52f3878700 1 -- 172.21.15.29:0/1782253200 shutdown_connections 2024-01-24T13:35:44.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.505+0000 7f52f3878700 1 --2- 172.21.15.29:0/1782253200 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f52ec106a90 0x7f52ec108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:44.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.505+0000 7f52f3878700 1 --2- 172.21.15.29:0/1782253200 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52ec101830 0x7f52ec103c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:44.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.505+0000 7f52f3878700 1 --2- 172.21.15.29:0/1782253200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52ec104160 0x7f52ec106550 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:44.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.505+0000 7f52f3878700 1 -- 172.21.15.29:0/1782253200 >> 172.21.15.29:0/1782253200 conn(0x7f52ec0fb1a0 msgr2=0x7f52ec0fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:44.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.505+0000 7f52f3878700 1 -- 172.21.15.29:0/1782253200 shutdown_connections 2024-01-24T13:35:44.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.505+0000 7f52f3878700 1 -- 172.21.15.29:0/1782253200 wait complete. 2024-01-24T13:35:44.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.506+0000 7f52f3878700 1 Processor -- start 2024-01-24T13:35:44.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.506+0000 7f52f3878700 1 -- start start 2024-01-24T13:35:44.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.506+0000 7f52f3878700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52ec101830 0x7f52ec196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:44.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.506+0000 7f52f3878700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52ec104160 0x7f52ec196f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:44.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.506+0000 7f52f3878700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f52ec106a90 0x7f52ec19dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:44.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.506+0000 7f52f3878700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ec19e580 con 0x7f52ec101830 2024-01-24T13:35:44.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.506+0000 7f52f3878700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ec19e6c0 con 0x7f52ec106a90 2024-01-24T13:35:44.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.506+0000 7f52f3878700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ec19e850 con 0x7f52ec104160 2024-01-24T13:35:44.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.506+0000 7f52f1614700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52ec101830 0x7f52ec196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:44.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.506+0000 7f52f1614700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52ec101830 0x7f52ec196a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44758/0 (socket says 172.21.15.29:44758) 2024-01-24T13:35:44.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.506+0000 7f52f1614700 1 -- 172.21.15.29:0/3952775416 learned_addr learned my addr 172.21.15.29:0/3952775416 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:44.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.506+0000 7f52f1e15700 1 --2- 172.21.15.29:0/3952775416 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f52ec106a90 0x7f52ec19dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:44.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.507+0000 7f52f0e13700 1 --2- 172.21.15.29:0/3952775416 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52ec104160 0x7f52ec196f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:44.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.507+0000 7f52f1614700 1 -- 172.21.15.29:0/3952775416 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52ec104160 msgr2=0x7f52ec196f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:44.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.507+0000 7f52f1614700 1 --2- 172.21.15.29:0/3952775416 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52ec104160 0x7f52ec196f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:44.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.507+0000 7f52f1614700 1 -- 172.21.15.29:0/3952775416 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f52ec106a90 msgr2=0x7f52ec19dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:44.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.507+0000 7f52f1614700 1 --2- 172.21.15.29:0/3952775416 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f52ec106a90 0x7f52ec19dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:44.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.507+0000 7f52f1614700 1 -- 172.21.15.29:0/3952775416 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52dc0095f0 con 0x7f52ec101830 2024-01-24T13:35:44.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.507+0000 7f52f1614700 1 --2- 172.21.15.29:0/3952775416 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52ec101830 0x7f52ec196a00 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f52e800d950 tx=0x7f52e800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:44.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.507+0000 7f52e27fc700 1 -- 172.21.15.29:0/3952775416 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e800f890 con 0x7f52ec101830 2024-01-24T13:35:44.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.507+0000 7f52f3878700 1 -- 172.21.15.29:0/3952775416 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52ec19eb30 con 0x7f52ec101830 2024-01-24T13:35:44.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.507+0000 7f52e27fc700 1 -- 172.21.15.29:0/3952775416 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f52e8010460 con 0x7f52ec101830 2024-01-24T13:35:44.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.507+0000 7f52e27fc700 1 -- 172.21.15.29:0/3952775416 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e800e610 con 0x7f52ec101830 2024-01-24T13:35:44.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.507+0000 7f52f3878700 1 -- 172.21.15.29:0/3952775416 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52ec19f040 con 0x7f52ec101830 2024-01-24T13:35:44.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.508+0000 7f52f3878700 1 -- 172.21.15.29:0/3952775416 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52ec10c510 con 0x7f52ec101830 2024-01-24T13:35:44.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.509+0000 7f52e27fc700 1 -- 172.21.15.29:0/3952775416 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f52e80108d0 con 0x7f52ec101830 2024-01-24T13:35:44.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.510+0000 7f52e27fc700 1 --2- 172.21.15.29:0/3952775416 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f52d80a0520 0x7f52d80a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:44.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.510+0000 7f52f0e13700 1 --2- 172.21.15.29:0/3952775416 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f52d80a0520 0x7f52d80a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:44.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.510+0000 7f52e27fc700 1 -- 172.21.15.29:0/3952775416 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f52e80cb930 con 0x7f52ec101830 2024-01-24T13:35:44.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.511+0000 7f52f0e13700 1 --2- 172.21.15.29:0/3952775416 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f52d80a0520 0x7f52d80a29e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f52dc00b5c0 tx=0x7f52dc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:44.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.513+0000 7f52e27fc700 1 -- 172.21.15.29:0/3952775416 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f52e80952e0 con 0x7f52ec101830 2024-01-24T13:35:44.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.724+0000 7f52f3878700 1 -- 172.21.15.29:0/3952775416 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f52ec19f9f0 con 0x7f52ec101830 2024-01-24T13:35:44.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.725+0000 7f52e27fc700 1 -- 172.21.15.29:0/3952775416 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7f52e8020020 con 0x7f52ec101830 2024-01-24T13:35:44.726 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-01-24T13:35:44.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.727+0000 7f52f3878700 1 -- 172.21.15.29:0/3952775416 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f52d80a0520 msgr2=0x7f52d80a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:44.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.727+0000 7f52f3878700 1 --2- 172.21.15.29:0/3952775416 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f52d80a0520 0x7f52d80a29e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f52dc00b5c0 tx=0x7f52dc005fd0 comp rx=0 tx=0).stop 2024-01-24T13:35:44.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.727+0000 7f52f3878700 1 -- 172.21.15.29:0/3952775416 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52ec101830 msgr2=0x7f52ec196a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:44.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.727+0000 7f52f3878700 1 --2- 172.21.15.29:0/3952775416 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52ec101830 0x7f52ec196a00 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f52e800d950 tx=0x7f52e800dc60 comp rx=0 tx=0).stop 2024-01-24T13:35:44.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.727+0000 7f52f3878700 1 -- 172.21.15.29:0/3952775416 shutdown_connections 2024-01-24T13:35:44.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.727+0000 7f52f3878700 1 --2- 172.21.15.29:0/3952775416 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f52ec106a90 0x7f52ec19dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:44.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.727+0000 7f52f3878700 1 --2- 172.21.15.29:0/3952775416 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f52d80a0520 0x7f52d80a29e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:44.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.727+0000 7f52f3878700 1 --2- 172.21.15.29:0/3952775416 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52ec101830 0x7f52ec196a00 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:44.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.727+0000 7f52f3878700 1 --2- 172.21.15.29:0/3952775416 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52ec104160 0x7f52ec196f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:44.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.727+0000 7f52f3878700 1 -- 172.21.15.29:0/3952775416 >> 172.21.15.29:0/3952775416 conn(0x7f52ec0fb1a0 msgr2=0x7f52ec102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:44.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.727+0000 7f52f3878700 1 -- 172.21.15.29:0/3952775416 shutdown_connections 2024-01-24T13:35:44.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.727+0000 7f52f3878700 1 -- 172.21.15.29:0/3952775416 wait complete. 2024-01-24T13:35:44.742 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T13:35:44.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.883+0000 7f87d4503700 1 -- 172.21.15.29:0/2387246972 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87cc0ffc30 msgr2=0x7f87cc100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:44.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.883+0000 7f87d4503700 1 --2- 172.21.15.29:0/2387246972 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87cc0ffc30 0x7f87cc100050 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f87c80099c0 tx=0x7f87c8009cd0 comp rx=0 tx=0).stop 2024-01-24T13:35:44.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.883+0000 7f87d4503700 1 -- 172.21.15.29:0/2387246972 shutdown_connections 2024-01-24T13:35:44.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.883+0000 7f87d4503700 1 --2- 172.21.15.29:0/2387246972 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f87cc1017c0 0x7f87cc108070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:44.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.883+0000 7f87d4503700 1 --2- 172.21.15.29:0/2387246972 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87cc0ffc30 0x7f87cc100050 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:44.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.883+0000 7f87d4503700 1 --2- 172.21.15.29:0/2387246972 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87cc100e20 0x7f87cc101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:44.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.883+0000 7f87d4503700 1 -- 172.21.15.29:0/2387246972 >> 172.21.15.29:0/2387246972 conn(0x7f87cc0fb1a0 msgr2=0x7f87cc0fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:44.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.883+0000 7f87d4503700 1 -- 172.21.15.29:0/2387246972 shutdown_connections 2024-01-24T13:35:44.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.883+0000 7f87d4503700 1 -- 172.21.15.29:0/2387246972 wait complete. 2024-01-24T13:35:44.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.884+0000 7f87d4503700 1 Processor -- start 2024-01-24T13:35:44.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.884+0000 7f87d4503700 1 -- start start 2024-01-24T13:35:44.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.884+0000 7f87d4503700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f87cc0ffc30 0x7f87cc068a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:44.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.884+0000 7f87d4503700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87cc100e20 0x7f87cc19b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:44.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.884+0000 7f87d4503700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87cc1017c0 0x7f87cc19ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:44.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.884+0000 7f87d4503700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87cc19e5a0 con 0x7f87cc100e20 2024-01-24T13:35:44.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.884+0000 7f87d4503700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87cc19e710 con 0x7f87cc1017c0 2024-01-24T13:35:44.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.884+0000 7f87d4503700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87cc19e8d0 con 0x7f87cc0ffc30 2024-01-24T13:35:44.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.884+0000 7f87d1a9e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87cc100e20 0x7f87cc19b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:44.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.884+0000 7f87d1a9e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87cc100e20 0x7f87cc19b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44784/0 (socket says 172.21.15.29:44784) 2024-01-24T13:35:44.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.884+0000 7f87d1a9e700 1 -- 172.21.15.29:0/4087832913 learned_addr learned my addr 172.21.15.29:0/4087832913 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:44.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.884+0000 7f87d229f700 1 --2- 172.21.15.29:0/4087832913 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f87cc0ffc30 0x7f87cc068a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:44.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.884+0000 7f87d2aa0700 1 --2- 172.21.15.29:0/4087832913 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87cc1017c0 0x7f87cc19ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:44.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.885+0000 7f87d1a9e700 1 -- 172.21.15.29:0/4087832913 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f87cc0ffc30 msgr2=0x7f87cc068a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:44.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.885+0000 7f87d1a9e700 1 --2- 172.21.15.29:0/4087832913 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f87cc0ffc30 0x7f87cc068a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:44.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.885+0000 7f87d1a9e700 1 -- 172.21.15.29:0/4087832913 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87cc1017c0 msgr2=0x7f87cc19ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:44.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.885+0000 7f87d1a9e700 1 --2- 172.21.15.29:0/4087832913 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87cc1017c0 0x7f87cc19ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:44.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.885+0000 7f87d1a9e700 1 -- 172.21.15.29:0/4087832913 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c80095f0 con 0x7f87cc100e20 2024-01-24T13:35:44.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.885+0000 7f87d2aa0700 1 --2- 172.21.15.29:0/4087832913 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87cc1017c0 0x7f87cc19ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T13:35:44.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.885+0000 7f87d1a9e700 1 --2- 172.21.15.29:0/4087832913 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87cc100e20 0x7f87cc19b780 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f87bc00d950 tx=0x7f87bc00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:44.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.885+0000 7f87c37fe700 1 -- 172.21.15.29:0/4087832913 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87bc009940 con 0x7f87cc100e20 2024-01-24T13:35:44.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.885+0000 7f87d4503700 1 -- 172.21.15.29:0/4087832913 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87cc19ebb0 con 0x7f87cc100e20 2024-01-24T13:35:44.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.885+0000 7f87c37fe700 1 -- 172.21.15.29:0/4087832913 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f87bc010460 con 0x7f87cc100e20 2024-01-24T13:35:44.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.885+0000 7f87c37fe700 1 -- 172.21.15.29:0/4087832913 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87bc00f620 con 0x7f87cc100e20 2024-01-24T13:35:44.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.885+0000 7f87d4503700 1 -- 172.21.15.29:0/4087832913 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87cc19f0c0 con 0x7f87cc100e20 2024-01-24T13:35:44.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.886+0000 7f87d4503700 1 -- 172.21.15.29:0/4087832913 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87cc107210 con 0x7f87cc100e20 2024-01-24T13:35:44.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.887+0000 7f87c37fe700 1 -- 172.21.15.29:0/4087832913 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f87bc0108d0 con 0x7f87cc100e20 2024-01-24T13:35:44.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.888+0000 7f87c37fe700 1 --2- 172.21.15.29:0/4087832913 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f87b80a0520 0x7f87b80a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:44.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.888+0000 7f87c37fe700 1 -- 172.21.15.29:0/4087832913 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f87bc0cb9a0 con 0x7f87cc100e20 2024-01-24T13:35:44.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.888+0000 7f87d229f700 1 --2- 172.21.15.29:0/4087832913 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f87b80a0520 0x7f87b80a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:44.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.889+0000 7f87d229f700 1 --2- 172.21.15.29:0/4087832913 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f87b80a0520 0x7f87b80a29e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f87c8009f50 tx=0x7f87c800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:44.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:44.892+0000 7f87c37fe700 1 -- 172.21.15.29:0/4087832913 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f87bc095350 con 0x7f87cc100e20 2024-01-24T13:35:45.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:44 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/2770959707' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-01-24T13:35:45.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:44 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/2278723833' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T13:35:45.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:44 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/3952775416' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-24T13:35:45.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.107+0000 7f87d4503700 1 -- 172.21.15.29:0/4087832913 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f87cc02ce00 con 0x7f87cc100e20 2024-01-24T13:35:45.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.107+0000 7f87c37fe700 1 -- 172.21.15.29:0/4087832913 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) v1 ==== 104+0+11 (secure 0 0 0) 0x7f87bc016020 con 0x7f87cc100e20 2024-01-24T13:35:45.108 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-01-24T13:35:45.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.109+0000 7f87d4503700 1 -- 172.21.15.29:0/4087832913 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f87b80a0520 msgr2=0x7f87b80a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:45.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.109+0000 7f87d4503700 1 --2- 172.21.15.29:0/4087832913 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f87b80a0520 0x7f87b80a29e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f87c8009f50 tx=0x7f87c800b540 comp rx=0 tx=0).stop 2024-01-24T13:35:45.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.109+0000 7f87d4503700 1 -- 172.21.15.29:0/4087832913 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87cc100e20 msgr2=0x7f87cc19b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:45.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.109+0000 7f87d4503700 1 --2- 172.21.15.29:0/4087832913 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87cc100e20 0x7f87cc19b780 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f87bc00d950 tx=0x7f87bc00dc60 comp rx=0 tx=0).stop 2024-01-24T13:35:45.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.110+0000 7f87d4503700 1 -- 172.21.15.29:0/4087832913 shutdown_connections 2024-01-24T13:35:45.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.110+0000 7f87d4503700 1 --2- 172.21.15.29:0/4087832913 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87cc1017c0 0x7f87cc19ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:45.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.110+0000 7f87d4503700 1 --2- 172.21.15.29:0/4087832913 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f87cc0ffc30 0x7f87cc068a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:45.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.110+0000 7f87d4503700 1 --2- 172.21.15.29:0/4087832913 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f87b80a0520 0x7f87b80a29e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:45.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.110+0000 7f87d4503700 1 --2- 172.21.15.29:0/4087832913 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87cc100e20 0x7f87cc19b780 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:45.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.110+0000 7f87d4503700 1 -- 172.21.15.29:0/4087832913 >> 172.21.15.29:0/4087832913 conn(0x7f87cc0fb1a0 msgr2=0x7f87cc104350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:45.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.110+0000 7f87d4503700 1 -- 172.21.15.29:0/4087832913 shutdown_connections 2024-01-24T13:35:45.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.110+0000 7f87d4503700 1 -- 172.21.15.29:0/4087832913 wait complete. 2024-01-24T13:35:45.124 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T13:35:45.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:44 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/2770959707' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-01-24T13:35:45.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:44 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/2278723833' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T13:35:45.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:44 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/3952775416' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-24T13:35:45.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.263+0000 7feeff8a5700 1 -- 172.21.15.29:0/1950833911 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feef80ffc30 msgr2=0x7feef8100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:45.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.263+0000 7feeff8a5700 1 --2- 172.21.15.29:0/1950833911 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feef80ffc30 0x7feef8100050 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7feef40099c0 tx=0x7feef4009cd0 comp rx=0 tx=0).stop 2024-01-24T13:35:45.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.263+0000 7feeff8a5700 1 -- 172.21.15.29:0/1950833911 shutdown_connections 2024-01-24T13:35:45.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.263+0000 7feeff8a5700 1 --2- 172.21.15.29:0/1950833911 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feef81017c0 0x7feef8108070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:45.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.263+0000 7feeff8a5700 1 --2- 172.21.15.29:0/1950833911 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feef80ffc30 0x7feef8100050 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:45.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.263+0000 7feeff8a5700 1 --2- 172.21.15.29:0/1950833911 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feef8100e20 0x7feef8101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:45.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.263+0000 7feeff8a5700 1 -- 172.21.15.29:0/1950833911 >> 172.21.15.29:0/1950833911 conn(0x7feef80fb1a0 msgr2=0x7feef80fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:45.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.263+0000 7feeff8a5700 1 -- 172.21.15.29:0/1950833911 shutdown_connections 2024-01-24T13:35:45.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.263+0000 7feeff8a5700 1 -- 172.21.15.29:0/1950833911 wait complete. 2024-01-24T13:35:45.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.264+0000 7feeff8a5700 1 Processor -- start 2024-01-24T13:35:45.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.264+0000 7feeff8a5700 1 -- start start 2024-01-24T13:35:45.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.264+0000 7feeff8a5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feef80ffc30 0x7feef8068a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:45.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.264+0000 7feeff8a5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feef8100e20 0x7feef819b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:45.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.264+0000 7feeff8a5700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feef81017c0 0x7feef819ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:45.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.264+0000 7feeff8a5700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef819e560 con 0x7feef8100e20 2024-01-24T13:35:45.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.264+0000 7feeff8a5700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef819e6d0 con 0x7feef81017c0 2024-01-24T13:35:45.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.264+0000 7feeff8a5700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef819e890 con 0x7feef80ffc30 2024-01-24T13:35:45.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.264+0000 7feefce40700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feef8100e20 0x7feef819b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:45.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.264+0000 7feefce40700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feef8100e20 0x7feef819b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44792/0 (socket says 172.21.15.29:44792) 2024-01-24T13:35:45.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.264+0000 7feefce40700 1 -- 172.21.15.29:0/674521514 learned_addr learned my addr 172.21.15.29:0/674521514 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:45.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.264+0000 7feefde42700 1 --2- 172.21.15.29:0/674521514 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feef81017c0 0x7feef819ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:45.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.264+0000 7feefd641700 1 --2- 172.21.15.29:0/674521514 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feef80ffc30 0x7feef8068a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:45.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.264+0000 7feefce40700 1 -- 172.21.15.29:0/674521514 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feef80ffc30 msgr2=0x7feef8068a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:45.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.264+0000 7feefce40700 1 --2- 172.21.15.29:0/674521514 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feef80ffc30 0x7feef8068a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:45.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.264+0000 7feefce40700 1 -- 172.21.15.29:0/674521514 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feef81017c0 msgr2=0x7feef819ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:45.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.264+0000 7feefce40700 1 --2- 172.21.15.29:0/674521514 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feef81017c0 0x7feef819ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:45.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.264+0000 7feefce40700 1 -- 172.21.15.29:0/674521514 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef40095f0 con 0x7feef8100e20 2024-01-24T13:35:45.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.264+0000 7feefd641700 1 --2- 172.21.15.29:0/674521514 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feef80ffc30 0x7feef8068a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T13:35:45.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.264+0000 7feefce40700 1 --2- 172.21.15.29:0/674521514 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feef8100e20 0x7feef819b780 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7feee800d980 tx=0x7feee800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:45.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.265+0000 7feeee7fc700 1 -- 172.21.15.29:0/674521514 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee8004220 con 0x7feef8100e20 2024-01-24T13:35:45.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.265+0000 7feeee7fc700 1 -- 172.21.15.29:0/674521514 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feee8007640 con 0x7feef8100e20 2024-01-24T13:35:45.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.265+0000 7feeff8a5700 1 -- 172.21.15.29:0/674521514 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feef819eb70 con 0x7feef8100e20 2024-01-24T13:35:45.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.265+0000 7feeee7fc700 1 -- 172.21.15.29:0/674521514 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee8005610 con 0x7feef8100e20 2024-01-24T13:35:45.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.265+0000 7feeff8a5700 1 -- 172.21.15.29:0/674521514 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feef819f0d0 con 0x7feef8100e20 2024-01-24T13:35:45.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.266+0000 7feeff8a5700 1 -- 172.21.15.29:0/674521514 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feef8106be0 con 0x7feef8100e20 2024-01-24T13:35:45.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.267+0000 7feeee7fc700 1 -- 172.21.15.29:0/674521514 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7feee8004750 con 0x7feef8100e20 2024-01-24T13:35:45.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.267+0000 7feeee7fc700 1 --2- 172.21.15.29:0/674521514 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7feee40a05a0 0x7feee40a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:45.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.268+0000 7feefd641700 1 --2- 172.21.15.29:0/674521514 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7feee40a05a0 0x7feee40a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:45.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.268+0000 7feeee7fc700 1 -- 172.21.15.29:0/674521514 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7feee80cb9b0 con 0x7feef8100e20 2024-01-24T13:35:45.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.268+0000 7feefd641700 1 --2- 172.21.15.29:0/674521514 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7feee40a05a0 0x7feee40a2a60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7feef400b580 tx=0x7feef4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:45.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.272+0000 7feeee7fc700 1 -- 172.21.15.29:0/674521514 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7feee8095360 con 0x7feef8100e20 2024-01-24T13:35:45.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:44 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/2770959707' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-01-24T13:35:45.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:44 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/2278723833' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T13:35:45.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:44 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/3952775416' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-24T13:35:45.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.485+0000 7feeff8a5700 1 -- 172.21.15.29:0/674521514 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7feef80619f0 con 0x7feef8100e20 2024-01-24T13:35:45.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.486+0000 7feeee7fc700 1 -- 172.21.15.29:0/674521514 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) v1 ==== 100+0+11 (secure 0 0 0) 0x7feee8016020 con 0x7feef8100e20 2024-01-24T13:35:45.487 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2024-01-24T13:35:45.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.488+0000 7feeff8a5700 1 -- 172.21.15.29:0/674521514 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7feee40a05a0 msgr2=0x7feee40a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:45.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.488+0000 7feeff8a5700 1 --2- 172.21.15.29:0/674521514 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7feee40a05a0 0x7feee40a2a60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7feef400b580 tx=0x7feef4005fb0 comp rx=0 tx=0).stop 2024-01-24T13:35:45.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.488+0000 7feeff8a5700 1 -- 172.21.15.29:0/674521514 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feef8100e20 msgr2=0x7feef819b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:45.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.488+0000 7feeff8a5700 1 --2- 172.21.15.29:0/674521514 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feef8100e20 0x7feef819b780 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7feee800d980 tx=0x7feee800dc90 comp rx=0 tx=0).stop 2024-01-24T13:35:45.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.488+0000 7feeff8a5700 1 -- 172.21.15.29:0/674521514 shutdown_connections 2024-01-24T13:35:45.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.488+0000 7feeff8a5700 1 --2- 172.21.15.29:0/674521514 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feef81017c0 0x7feef819ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:45.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.488+0000 7feeff8a5700 1 --2- 172.21.15.29:0/674521514 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feef80ffc30 0x7feef8068a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:45.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.488+0000 7feeff8a5700 1 --2- 172.21.15.29:0/674521514 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7feee40a05a0 0x7feee40a2a60 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:45.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.488+0000 7feeff8a5700 1 --2- 172.21.15.29:0/674521514 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feef8100e20 0x7feef819b780 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:45.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.488+0000 7feeff8a5700 1 -- 172.21.15.29:0/674521514 >> 172.21.15.29:0/674521514 conn(0x7feef80fb1a0 msgr2=0x7feef80fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:45.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.488+0000 7feeff8a5700 1 -- 172.21.15.29:0/674521514 shutdown_connections 2024-01-24T13:35:45.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.488+0000 7feeff8a5700 1 -- 172.21.15.29:0/674521514 wait complete. 2024-01-24T13:35:45.503 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-01-24T13:35:45.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.641+0000 7f2ec44fb700 1 -- 172.21.15.29:0/2245768264 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ebc0ffc30 msgr2=0x7f2ebc100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:45.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.641+0000 7f2ec44fb700 1 --2- 172.21.15.29:0/2245768264 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ebc0ffc30 0x7f2ebc100050 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2eb80099c0 tx=0x7f2eb8009cd0 comp rx=0 tx=0).stop 2024-01-24T13:35:45.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.641+0000 7f2ec44fb700 1 -- 172.21.15.29:0/2245768264 shutdown_connections 2024-01-24T13:35:45.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.641+0000 7f2ec44fb700 1 --2- 172.21.15.29:0/2245768264 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2ebc1017c0 0x7f2ebc108070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:45.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.641+0000 7f2ec44fb700 1 --2- 172.21.15.29:0/2245768264 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ebc0ffc30 0x7f2ebc100050 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:45.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.641+0000 7f2ec44fb700 1 --2- 172.21.15.29:0/2245768264 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ebc100e20 0x7f2ebc101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:45.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.641+0000 7f2ec44fb700 1 -- 172.21.15.29:0/2245768264 >> 172.21.15.29:0/2245768264 conn(0x7f2ebc0fb1a0 msgr2=0x7f2ebc0fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:45.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.641+0000 7f2ec44fb700 1 -- 172.21.15.29:0/2245768264 shutdown_connections 2024-01-24T13:35:45.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.641+0000 7f2ec44fb700 1 -- 172.21.15.29:0/2245768264 wait complete. 2024-01-24T13:35:45.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.642+0000 7f2ec44fb700 1 Processor -- start 2024-01-24T13:35:45.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.642+0000 7f2ec44fb700 1 -- start start 2024-01-24T13:35:45.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.642+0000 7f2ec44fb700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ebc0ffc30 0x7f2ebc103bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:45.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.642+0000 7f2ec44fb700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2ebc100e20 0x7f2ebc102200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:45.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.642+0000 7f2ec44fb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ebc1017c0 0x7f2ebc102740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:45.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.642+0000 7f2ec44fb700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ebc102ee0 con 0x7f2ebc1017c0 2024-01-24T13:35:45.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.642+0000 7f2ec44fb700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ebc103050 con 0x7f2ebc100e20 2024-01-24T13:35:45.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.642+0000 7f2ec44fb700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ebc1a0aa0 con 0x7f2ebc0ffc30 2024-01-24T13:35:45.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.642+0000 7f2ec2a98700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ebc1017c0 0x7f2ebc102740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:45.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.642+0000 7f2ec2a98700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ebc1017c0 0x7f2ebc102740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44798/0 (socket says 172.21.15.29:44798) 2024-01-24T13:35:45.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.642+0000 7f2ec2a98700 1 -- 172.21.15.29:0/823435214 learned_addr learned my addr 172.21.15.29:0/823435214 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:45.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.642+0000 7f2ec2297700 1 --2- 172.21.15.29:0/823435214 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ebc0ffc30 0x7f2ebc103bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:45.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.642+0000 7f2ec1a96700 1 --2- 172.21.15.29:0/823435214 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2ebc100e20 0x7f2ebc102200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:45.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.642+0000 7f2ec2a98700 1 -- 172.21.15.29:0/823435214 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ebc0ffc30 msgr2=0x7f2ebc103bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:45.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.642+0000 7f2ec2a98700 1 --2- 172.21.15.29:0/823435214 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ebc0ffc30 0x7f2ebc103bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:45.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.642+0000 7f2ec2a98700 1 -- 172.21.15.29:0/823435214 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2ebc100e20 msgr2=0x7f2ebc102200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:45.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.642+0000 7f2ec2a98700 1 --2- 172.21.15.29:0/823435214 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2ebc100e20 0x7f2ebc102200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:45.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.642+0000 7f2ec2a98700 1 -- 172.21.15.29:0/823435214 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb80095f0 con 0x7f2ebc1017c0 2024-01-24T13:35:45.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.642+0000 7f2ec2297700 1 --2- 172.21.15.29:0/823435214 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ebc0ffc30 0x7f2ebc103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T13:35:45.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.642+0000 7f2ec2a98700 1 --2- 172.21.15.29:0/823435214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ebc1017c0 0x7f2ebc102740 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2eb4007ee0 tx=0x7f2eb400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:45.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.643+0000 7f2eb37fe700 1 -- 172.21.15.29:0/823435214 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb4017070 con 0x7f2ebc1017c0 2024-01-24T13:35:45.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.643+0000 7f2eb37fe700 1 -- 172.21.15.29:0/823435214 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2eb400f040 con 0x7f2ebc1017c0 2024-01-24T13:35:45.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.643+0000 7f2ec44fb700 1 -- 172.21.15.29:0/823435214 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ebc1a0d80 con 0x7f2ebc1017c0 2024-01-24T13:35:45.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.643+0000 7f2eb37fe700 1 -- 172.21.15.29:0/823435214 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb4005740 con 0x7f2ebc1017c0 2024-01-24T13:35:45.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.643+0000 7f2ec44fb700 1 -- 172.21.15.29:0/823435214 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ebc1a1230 con 0x7f2ebc1017c0 2024-01-24T13:35:45.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.644+0000 7f2ec44fb700 1 -- 172.21.15.29:0/823435214 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ebc10b440 con 0x7f2ebc1017c0 2024-01-24T13:35:45.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.645+0000 7f2eb37fe700 1 -- 172.21.15.29:0/823435214 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f2eb40058a0 con 0x7f2ebc1017c0 2024-01-24T13:35:45.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.646+0000 7f2eb37fe700 1 --2- 172.21.15.29:0/823435214 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f2ea80a05a0 0x7f2ea80a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:45.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.646+0000 7f2ec2297700 1 --2- 172.21.15.29:0/823435214 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f2ea80a05a0 0x7f2ea80a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:45.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.646+0000 7f2eb37fe700 1 -- 172.21.15.29:0/823435214 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f2eb40cb390 con 0x7f2ebc1017c0 2024-01-24T13:35:45.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.646+0000 7f2ec2297700 1 --2- 172.21.15.29:0/823435214 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f2ea80a05a0 0x7f2ea80a2a60 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2eb800b580 tx=0x7f2eb8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:45.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.650+0000 7f2eb37fe700 1 -- 172.21.15.29:0/823435214 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f2eb4094ce0 con 0x7f2ebc1017c0 2024-01-24T13:35:45.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.863+0000 7f2ec44fb700 1 -- 172.21.15.29:0/823435214 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f2ebc1a1840 con 0x7f2ebc1017c0 2024-01-24T13:35:45.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.864+0000 7f2eb37fe700 1 -- 172.21.15.29:0/823435214 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v85) v1 ==== 103+0+11 (secure 0 0 0) 0x7f2eb4098300 con 0x7f2ebc1017c0 2024-01-24T13:35:45.865 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-01-24T13:35:45.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.866+0000 7f2ec44fb700 1 -- 172.21.15.29:0/823435214 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f2ea80a05a0 msgr2=0x7f2ea80a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:45.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.866+0000 7f2ec44fb700 1 --2- 172.21.15.29:0/823435214 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f2ea80a05a0 0x7f2ea80a2a60 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2eb800b580 tx=0x7f2eb8005fb0 comp rx=0 tx=0).stop 2024-01-24T13:35:45.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.866+0000 7f2ec44fb700 1 -- 172.21.15.29:0/823435214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ebc1017c0 msgr2=0x7f2ebc102740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:45.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.866+0000 7f2ec44fb700 1 --2- 172.21.15.29:0/823435214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ebc1017c0 0x7f2ebc102740 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2eb4007ee0 tx=0x7f2eb400d3b0 comp rx=0 tx=0).stop 2024-01-24T13:35:45.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.866+0000 7f2ec44fb700 1 -- 172.21.15.29:0/823435214 shutdown_connections 2024-01-24T13:35:45.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.866+0000 7f2ec44fb700 1 --2- 172.21.15.29:0/823435214 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f2ea80a05a0 0x7f2ea80a2a60 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:45.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.866+0000 7f2ec44fb700 1 --2- 172.21.15.29:0/823435214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ebc1017c0 0x7f2ebc102740 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:45.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.866+0000 7f2ec44fb700 1 --2- 172.21.15.29:0/823435214 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ebc0ffc30 0x7f2ebc103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:45.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.866+0000 7f2ec44fb700 1 --2- 172.21.15.29:0/823435214 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2ebc100e20 0x7f2ebc102200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:45.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.866+0000 7f2ec44fb700 1 -- 172.21.15.29:0/823435214 >> 172.21.15.29:0/823435214 conn(0x7f2ebc0fb1a0 msgr2=0x7f2ebc067ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:45.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.866+0000 7f2ec44fb700 1 -- 172.21.15.29:0/823435214 shutdown_connections 2024-01-24T13:35:45.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:45.866+0000 7f2ec44fb700 1 -- 172.21.15.29:0/823435214 wait complete. 2024-01-24T13:35:45.880 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T13:35:46.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.019+0000 7f5090fc2700 1 -- 172.21.15.29:0/2371417115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f508c1017f0 msgr2=0x7f508c1080a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:46.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.019+0000 7f5090fc2700 1 --2- 172.21.15.29:0/2371417115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f508c1017f0 0x7f508c1080a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f508400b0d0 tx=0x7f508400b3e0 comp rx=0 tx=0).stop 2024-01-24T13:35:46.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.019+0000 7f5090fc2700 1 -- 172.21.15.29:0/2371417115 shutdown_connections 2024-01-24T13:35:46.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.019+0000 7f5090fc2700 1 --2- 172.21.15.29:0/2371417115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f508c1017f0 0x7f508c1080a0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:46.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.019+0000 7f5090fc2700 1 --2- 172.21.15.29:0/2371417115 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f508c0ffc60 0x7f508c100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:46.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.019+0000 7f5090fc2700 1 --2- 172.21.15.29:0/2371417115 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f508c100e50 0x7f508c1012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:46.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.019+0000 7f5090fc2700 1 -- 172.21.15.29:0/2371417115 >> 172.21.15.29:0/2371417115 conn(0x7f508c0fb1d0 msgr2=0x7f508c0fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:46.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.019+0000 7f5090fc2700 1 -- 172.21.15.29:0/2371417115 shutdown_connections 2024-01-24T13:35:46.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.019+0000 7f5090fc2700 1 -- 172.21.15.29:0/2371417115 wait complete. 2024-01-24T13:35:46.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.020+0000 7f5090fc2700 1 Processor -- start 2024-01-24T13:35:46.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.020+0000 7f5090fc2700 1 -- start start 2024-01-24T13:35:46.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.021+0000 7f5090fc2700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f508c0ffc60 0x7f508c195ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:46.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.021+0000 7f5090fc2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f508c100e50 0x7f508c195ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:46.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.021+0000 7f5090fc2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f508c1017f0 0x7f508c19d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:46.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.021+0000 7f5090fc2700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f508c19d630 con 0x7f508c1017f0 2024-01-24T13:35:46.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.021+0000 7f5090fc2700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f508c19d770 con 0x7f508c0ffc60 2024-01-24T13:35:46.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.021+0000 7f5090fc2700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f508c19d900 con 0x7f508c100e50 2024-01-24T13:35:46.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.021+0000 7f508ad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f508c1017f0 0x7f508c19d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:46.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.021+0000 7f508ad9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f508c1017f0 0x7f508c19d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44812/0 (socket says 172.21.15.29:44812) 2024-01-24T13:35:46.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.021+0000 7f508ad9d700 1 -- 172.21.15.29:0/1465309573 learned_addr learned my addr 172.21.15.29:0/1465309573 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:46.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.021+0000 7f5089d9b700 1 --2- 172.21.15.29:0/1465309573 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f508c100e50 0x7f508c195ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:46.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.021+0000 7f508a59c700 1 --2- 172.21.15.29:0/1465309573 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f508c0ffc60 0x7f508c195ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:46.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.021+0000 7f508ad9d700 1 -- 172.21.15.29:0/1465309573 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f508c100e50 msgr2=0x7f508c195ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:46.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.021+0000 7f508ad9d700 1 --2- 172.21.15.29:0/1465309573 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f508c100e50 0x7f508c195ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:46.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.021+0000 7f508ad9d700 1 -- 172.21.15.29:0/1465309573 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f508c0ffc60 msgr2=0x7f508c195ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:46.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.021+0000 7f508ad9d700 1 --2- 172.21.15.29:0/1465309573 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f508c0ffc60 0x7f508c195ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:46.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.021+0000 7f508ad9d700 1 -- 172.21.15.29:0/1465309573 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5084009d00 con 0x7f508c1017f0 2024-01-24T13:35:46.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.021+0000 7f508a59c700 1 --2- 172.21.15.29:0/1465309573 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f508c0ffc60 0x7f508c195ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T13:35:46.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.021+0000 7f508ad9d700 1 --2- 172.21.15.29:0/1465309573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f508c1017f0 0x7f508c19d060 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f508400b820 tx=0x7f5084007930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:46.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.022+0000 7f507b7fe700 1 -- 172.21.15.29:0/1465309573 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508400e040 con 0x7f508c1017f0 2024-01-24T13:35:46.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.022+0000 7f507b7fe700 1 -- 172.21.15.29:0/1465309573 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5084003c50 con 0x7f508c1017f0 2024-01-24T13:35:46.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.022+0000 7f507b7fe700 1 -- 172.21.15.29:0/1465309573 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508401d6e0 con 0x7f508c1017f0 2024-01-24T13:35:46.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.022+0000 7f5090fc2700 1 -- 172.21.15.29:0/1465309573 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f508c19db80 con 0x7f508c1017f0 2024-01-24T13:35:46.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.022+0000 7f5090fc2700 1 -- 172.21.15.29:0/1465309573 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f508c19df50 con 0x7f508c1017f0 2024-01-24T13:35:46.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.023+0000 7f5090fc2700 1 -- 172.21.15.29:0/1465309573 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f508c10b540 con 0x7f508c1017f0 2024-01-24T13:35:46.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.025+0000 7f507b7fe700 1 -- 172.21.15.29:0/1465309573 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f50840050d0 con 0x7f508c1017f0 2024-01-24T13:35:46.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.025+0000 7f507b7fe700 1 --2- 172.21.15.29:0/1465309573 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f50740a0520 0x7f50740a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:46.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.025+0000 7f507b7fe700 1 -- 172.21.15.29:0/1465309573 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f50840ccc10 con 0x7f508c1017f0 2024-01-24T13:35:46.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.025+0000 7f508a59c700 1 --2- 172.21.15.29:0/1465309573 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f50740a0520 0x7f50740a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:46.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.026+0000 7f508a59c700 1 --2- 172.21.15.29:0/1465309573 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f50740a0520 0x7f50740a29e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f508c100cb0 tx=0x7f507c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:46.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.029+0000 7f507b7fe700 1 -- 172.21.15.29:0/1465309573 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f50840966e0 con 0x7f508c1017f0 2024-01-24T13:35:46.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:45 smithi163 ceph-mon[44969]: pgmap v232: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 921 B/s wr, 3 op/s 2024-01-24T13:35:46.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:45 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/4087832913' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-01-24T13:35:46.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:45 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/674521514' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-01-24T13:35:46.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.256+0000 7f5090fc2700 1 -- 172.21.15.29:0/1465309573 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f508c106b60 con 0x7f508c1017f0 2024-01-24T13:35:46.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.257+0000 7f507b7fe700 1 -- 172.21.15.29:0/1465309573 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v85) v1 ==== 81+0+587 (secure 0 0 0) 0x7f5084099d00 con 0x7f508c1017f0 2024-01-24T13:35:46.258 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/1419474286 2024-01-25T13:30:30.728638+0000 2024-01-24T13:35:46.258 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6800/835547951 2024-01-25T13:30:08.775587+0000 2024-01-24T13:35:46.258 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6801/1996568496 2024-01-25T13:30:30.728638+0000 2024-01-24T13:35:46.258 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/2727527401 2024-01-25T13:30:08.775587+0000 2024-01-24T13:35:46.258 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/41509684 2024-01-25T13:30:30.728638+0000 2024-01-24T13:35:46.258 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6800/1996568496 2024-01-25T13:30:30.728638+0000 2024-01-24T13:35:46.258 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/2833864599 2024-01-25T13:30:30.728638+0000 2024-01-24T13:35:46.258 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/168349482 2024-01-25T13:30:08.775587+0000 2024-01-24T13:35:46.258 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6801/835547951 2024-01-25T13:30:08.775587+0000 2024-01-24T13:35:46.259 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/1142561996 2024-01-25T13:30:08.775587+0000 2024-01-24T13:35:46.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.259+0000 7f5090fc2700 1 -- 172.21.15.29:0/1465309573 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f50740a0520 msgr2=0x7f50740a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:46.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.259+0000 7f5090fc2700 1 --2- 172.21.15.29:0/1465309573 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f50740a0520 0x7f50740a29e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f508c100cb0 tx=0x7f507c005f90 comp rx=0 tx=0).stop 2024-01-24T13:35:46.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.259+0000 7f5090fc2700 1 -- 172.21.15.29:0/1465309573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f508c1017f0 msgr2=0x7f508c19d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:46.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.259+0000 7f5090fc2700 1 --2- 172.21.15.29:0/1465309573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f508c1017f0 0x7f508c19d060 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f508400b820 tx=0x7f5084007930 comp rx=0 tx=0).stop 2024-01-24T13:35:46.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.259+0000 7f5090fc2700 1 -- 172.21.15.29:0/1465309573 shutdown_connections 2024-01-24T13:35:46.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.259+0000 7f5090fc2700 1 --2- 172.21.15.29:0/1465309573 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f50740a0520 0x7f50740a29e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:46.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.259+0000 7f5090fc2700 1 --2- 172.21.15.29:0/1465309573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f508c1017f0 0x7f508c19d060 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:46.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.259+0000 7f5090fc2700 1 --2- 172.21.15.29:0/1465309573 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f508c0ffc60 0x7f508c195ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:46.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.259+0000 7f5090fc2700 1 --2- 172.21.15.29:0/1465309573 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f508c100e50 0x7f508c195ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:46.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.259+0000 7f5090fc2700 1 -- 172.21.15.29:0/1465309573 >> 172.21.15.29:0/1465309573 conn(0x7f508c0fb1d0 msgr2=0x7f508c0fd410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:46.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.260+0000 7f5090fc2700 1 -- 172.21.15.29:0/1465309573 shutdown_connections 2024-01-24T13:35:46.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.260+0000 7f5090fc2700 1 -- 172.21.15.29:0/1465309573 wait complete. 2024-01-24T13:35:46.262 INFO:teuthology.orchestra.run.smithi029.stderr:listed 10 entries 2024-01-24T13:35:46.273 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:45 smithi029 ceph-mon[40651]: pgmap v232: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 921 B/s wr, 3 op/s 2024-01-24T13:35:46.273 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:45 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/4087832913' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-01-24T13:35:46.273 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:45 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/674521514' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-01-24T13:35:46.274 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi029.front.sepia.ceph.com... 2024-01-24T13:35:46.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T13:35:46.274 DEBUG:teuthology.orchestra.run.smithi029:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-01-24T13:35:46.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:45 smithi187 ceph-mon[44911]: pgmap v232: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 921 B/s wr, 3 op/s 2024-01-24T13:35:46.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:45 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/4087832913' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-01-24T13:35:46.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:45 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/674521514' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-01-24T13:35:46.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi029.stderr:ceph-fuse[94890]: fuse finished with error 0 and tester_r 0 2024-01-24T13:35:46.310 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T13:35:46.399 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2024-01-24T13:35:46.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T13:35:46.400 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-01-24T13:35:46.434 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T13:35:46.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.619+0000 7faef3e44700 1 -- 172.21.15.29:0/687943983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faeec1017b0 msgr2=0x7faeec108060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:46.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.619+0000 7faef3e44700 1 --2- 172.21.15.29:0/687943983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faeec1017b0 0x7faeec108060 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7faee400b0d0 tx=0x7faee400b3e0 comp rx=0 tx=0).stop 2024-01-24T13:35:46.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.619+0000 7faef3e44700 1 -- 172.21.15.29:0/687943983 shutdown_connections 2024-01-24T13:35:46.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.619+0000 7faef3e44700 1 --2- 172.21.15.29:0/687943983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faeec1017b0 0x7faeec108060 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:46.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.619+0000 7faef3e44700 1 --2- 172.21.15.29:0/687943983 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faeec0ffc20 0x7faeec100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:46.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.619+0000 7faef3e44700 1 --2- 172.21.15.29:0/687943983 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faeec100e10 0x7faeec101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:46.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.619+0000 7faef3e44700 1 -- 172.21.15.29:0/687943983 >> 172.21.15.29:0/687943983 conn(0x7faeec0fb170 msgr2=0x7faeec0fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:46.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.619+0000 7faef3e44700 1 -- 172.21.15.29:0/687943983 shutdown_connections 2024-01-24T13:35:46.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.619+0000 7faef3e44700 1 -- 172.21.15.29:0/687943983 wait complete. 2024-01-24T13:35:46.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.620+0000 7faef3e44700 1 Processor -- start 2024-01-24T13:35:46.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.620+0000 7faef3e44700 1 -- start start 2024-01-24T13:35:46.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.620+0000 7faef3e44700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faeec0ffc20 0x7faeec196a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:46.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.620+0000 7faef3e44700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faeec100e10 0x7faeec196fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:46.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.620+0000 7faef3e44700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faeec1017b0 0x7faeec19e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:46.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.620+0000 7faef3e44700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeec19e5f0 con 0x7faeec1017b0 2024-01-24T13:35:46.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.620+0000 7faef3e44700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeec19e730 con 0x7faeec100e10 2024-01-24T13:35:46.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.620+0000 7faef3e44700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeec19e8c0 con 0x7faeec0ffc20 2024-01-24T13:35:46.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.620+0000 7faef23e1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faeec1017b0 0x7faeec19e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:46.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.620+0000 7faef23e1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faeec1017b0 0x7faeec19e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44836/0 (socket says 172.21.15.29:44836) 2024-01-24T13:35:46.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.620+0000 7faef13df700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faeec100e10 0x7faeec196fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:46.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.620+0000 7faef23e1700 1 -- 172.21.15.29:0/3564894799 learned_addr learned my addr 172.21.15.29:0/3564894799 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:46.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.620+0000 7faef1be0700 1 --2- 172.21.15.29:0/3564894799 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faeec0ffc20 0x7faeec196a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:46.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.621+0000 7faef23e1700 1 -- 172.21.15.29:0/3564894799 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faeec0ffc20 msgr2=0x7faeec196a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:46.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.621+0000 7faef23e1700 1 --2- 172.21.15.29:0/3564894799 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faeec0ffc20 0x7faeec196a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:46.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.621+0000 7faef23e1700 1 -- 172.21.15.29:0/3564894799 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faeec100e10 msgr2=0x7faeec196fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:46.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.621+0000 7faef23e1700 1 --2- 172.21.15.29:0/3564894799 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faeec100e10 0x7faeec196fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:46.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.621+0000 7faef23e1700 1 -- 172.21.15.29:0/3564894799 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee4009d00 con 0x7faeec1017b0 2024-01-24T13:35:46.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.621+0000 7faef13df700 1 --2- 172.21.15.29:0/3564894799 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faeec100e10 0x7faeec196fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T13:35:46.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.621+0000 7faef23e1700 1 --2- 172.21.15.29:0/3564894799 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faeec1017b0 0x7faeec19e020 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7faee4008f10 tx=0x7faee4008f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:46.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.621+0000 7faee2ffd700 1 -- 172.21.15.29:0/3564894799 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee400e040 con 0x7faeec1017b0 2024-01-24T13:35:46.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.621+0000 7faef3e44700 1 -- 172.21.15.29:0/3564894799 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeec102030 con 0x7faeec1017b0 2024-01-24T13:35:46.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.621+0000 7faee2ffd700 1 -- 172.21.15.29:0/3564894799 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faee401c6c0 con 0x7faeec1017b0 2024-01-24T13:35:46.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.621+0000 7faee2ffd700 1 -- 172.21.15.29:0/3564894799 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee40052d0 con 0x7faeec1017b0 2024-01-24T13:35:46.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.621+0000 7faef3e44700 1 -- 172.21.15.29:0/3564894799 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeec1024e0 con 0x7faeec1017b0 2024-01-24T13:35:46.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.622+0000 7faef3e44700 1 -- 172.21.15.29:0/3564894799 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeec10b440 con 0x7faeec1017b0 2024-01-24T13:35:46.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.623+0000 7faee2ffd700 1 -- 172.21.15.29:0/3564894799 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7faee4022070 con 0x7faeec1017b0 2024-01-24T13:35:46.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.624+0000 7faee2ffd700 1 --2- 172.21.15.29:0/3564894799 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7faed80a0570 0x7faed80a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:46.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.624+0000 7faee2ffd700 1 -- 172.21.15.29:0/3564894799 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7faee40d0e30 con 0x7faeec1017b0 2024-01-24T13:35:46.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.624+0000 7faef1be0700 1 --2- 172.21.15.29:0/3564894799 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7faed80a0570 0x7faed80a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:46.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.624+0000 7faef1be0700 1 --2- 172.21.15.29:0/3564894799 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7faed80a0570 0x7faed80a2a30 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7faeec100c70 tx=0x7faee8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:46.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.628+0000 7faee2ffd700 1 -- 172.21.15.29:0/3564894799 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7faee409a870 con 0x7faeec1017b0 2024-01-24T13:35:46.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.844+0000 7faef3e44700 1 -- 172.21.15.29:0/3564894799 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faeec10b630 con 0x7faeec1017b0 2024-01-24T13:35:46.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.846+0000 7faee2ffd700 1 -- 172.21.15.29:0/3564894799 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23981 (secure 0 0 0) 0x7faee409de90 con 0x7faeec1017b0 2024-01-24T13:35:46.847 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:35:46.847 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":85,"fsid":"83afcf5e-babc-11ee-95b1-87774f69a715","created":"2024-01-24T13:29:33.175895+0000","modified":"2024-01-24T13:35:34.866792+0000","last_up_change":"2024-01-24T13:34:57.141902+0000","last_in_change":"2024-01-24T13:34:41.071943+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T13:32:21.264634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-24T13:35:28.761733+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-24T13:35:29.763293+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-01-24T13:35:32.858844+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9857a607-2e6f-41a5-a86a-795970be6210","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":2406629076},{"type":"v1","addr":"172.21.15.29:6803","nonce":2406629076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":2406629076},{"type":"v1","addr":"172.21.15.29:6805","nonce":2406629076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2406629076},{"type":"v1","addr":"172.21.15.29:6809","nonce":2406629076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":2406629076},{"type":"v1","addr":"172.21.15.29:6807","nonce":2406629076}]},"public_addr":"172.21.15.29:6803/2406629076","cluster_addr":"172.21.15.29:6805/2406629076","heartbeat_back_addr":"172.21.15.29:6809/2406629076","heartbeat_front_addr":"172.21.15.29:6807/2406629076","state":["exists","up"]},{"osd":1,"uuid":"8bb2ba48-23d4-4bbd-a912-ae74f9e7f6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":2217244566},{"type":"v1","addr":"172.21.15.163:6801","nonce":2217244566}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":2217244566},{"type":"v1","addr":"172.21.15.163:6803","nonce":2217244566}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":2217244566},{"type":"v1","addr":"172.21.15.163:6807","nonce":2217244566}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":2217244566},{"type":"v1","addr":"172.21.15.163:6805","nonce":2217244566}]},"public_addr":"172.21.15.163:6801/2217244566","cluster_addr":"172.21.15.163:6803/2217244566","heartbeat_back_addr":"172.21.15.163:6807/2217244566","heartbeat_front_addr":"172.21.15.163:6805/2217244566","state":["exists","up"]},{"osd":2,"uuid":"f0ed4131-cd29-4706-89ea-fde40f46ea83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3589694992},{"type":"v1","addr":"172.21.15.187:6801","nonce":3589694992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3589694992},{"type":"v1","addr":"172.21.15.187:6803","nonce":3589694992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3589694992},{"type":"v1","addr":"172.21.15.187:6807","nonce":3589694992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3589694992},{"type":"v1","addr":"172.21.15.187:6805","nonce":3589694992}]},"public_addr":"172.21.15.187:6801/3589694992","cluster_addr":"172.21.15.187:6803/3589694992","heartbeat_back_addr":"172.21.15.187:6807/3589694992","heartbeat_front_addr":"172.21.15.187:6805/3589694992","state":["exists","up"]},{"osd":3,"uuid":"da821310-b502-4b77-a5ce-0ffed0a3dafc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3417210415},{"type":"v1","addr":"172.21.15.29:6811","nonce":3417210415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":3417210415},{"type":"v1","addr":"172.21.15.29:6813","nonce":3417210415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3417210415},{"type":"v1","addr":"172.21.15.29:6817","nonce":3417210415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3417210415},{"type":"v1","addr":"172.21.15.29:6815","nonce":3417210415}]},"public_addr":"172.21.15.29:6811/3417210415","cluster_addr":"172.21.15.29:6813/3417210415","heartbeat_back_addr":"172.21.15.29:6817/3417210415","heartbeat_front_addr":"172.21.15.29:6815/3417210415","state":["exists","up"]},{"osd":4,"uuid":"8b3a348f-e7dd-4ccb-82a5-2d3f5d101da8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3283787183},{"type":"v1","addr":"172.21.15.163:6809","nonce":3283787183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":3283787183},{"type":"v1","addr":"172.21.15.163:6811","nonce":3283787183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3283787183},{"type":"v1","addr":"172.21.15.163:6815","nonce":3283787183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":3283787183},{"type":"v1","addr":"172.21.15.163:6813","nonce":3283787183}]},"public_addr":"172.21.15.163:6809/3283787183","cluster_addr":"172.21.15.163:6811/3283787183","heartbeat_back_addr":"172.21.15.163:6815/3283787183","heartbeat_front_addr":"172.21.15.163:6813/3283787183","state":["exists","up"]},{"osd":5,"uuid":"c26c7155-618b-4712-acdb-d4fd6c8b7f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":2259228817},{"type":"v1","addr":"172.21.15.187:6809","nonce":2259228817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":2259228817},{"type":"v1","addr":"172.21.15.187:6811","nonce":2259228817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":2259228817},{"type":"v1","addr":"172.21.15.187:6815","nonce":2259228817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":2259228817},{"type":"v1","addr":"172.21.15.187:6813","nonce":2259228817}]},"public_addr":"172.21.15.187:6809/2259228817","cluster_addr":"172.21.15.187:6811/2259228817","heartbeat_back_addr":"172.21.15.187:6815/2259228817","heartbeat_front_addr":"172.21.15.187:6813/2259228817","state":["exists","up"]},{"osd":6,"uuid":"042ac0b4-1fe1-444f-bf51-ad89613a2185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":938400701},{"type":"v1","addr":"172.21.15.29:6819","nonce":938400701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":938400701},{"type":"v1","addr":"172.21.15.29:6821","nonce":938400701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":938400701},{"type":"v1","addr":"172.21.15.29:6825","nonce":938400701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":938400701},{"type":"v1","addr":"172.21.15.29:6823","nonce":938400701}]},"public_addr":"172.21.15.29:6819/938400701","cluster_addr":"172.21.15.29:6821/938400701","heartbeat_back_addr":"172.21.15.29:6825/938400701","heartbeat_front_addr":"172.21.15.29:6823/938400701","state":["exists","up"]},{"osd":7,"uuid":"add3d09b-76f3-4a93-83a2-30ac94c3fad1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2783976092},{"type":"v1","addr":"172.21.15.163:6817","nonce":2783976092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2783976092},{"type":"v1","addr":"172.21.15.163:6819","nonce":2783976092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2783976092},{"type":"v1","addr":"172.21.15.163:6823","nonce":2783976092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2783976092},{"type":"v1","addr":"172.21.15.163:6821","nonce":2783976092}]},"public_addr":"172.21.15.163:6817/2783976092","cluster_addr":"172.21.15.163:6819/2783976092","heartbeat_back_addr":"172.21.15.163:6823/2783976092","heartbeat_front_addr":"172.21.15.163:6821/2783976092","state":["exists","up"]},{"osd":8,"uuid":"71044d6a-b55f-4bcf-94c5-79f71eb7b422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":2856803882},{"type":"v1","addr":"172.21.15.187:6817","nonce":2856803882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":2856803882},{"type":"v1","addr":"172.21.15.187:6819","nonce":2856803882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":2856803882},{"type":"v1","addr":"172.21.15.187:6823","nonce":2856803882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":2856803882},{"type":"v1","addr":"172.21.15.187:6821","nonce":2856803882}]},"public_addr":"172.21.15.187:6817/2856803882","cluster_addr":"172.21.15.187:6819/2856803882","heartbeat_back_addr":"172.21.15.187:6823/2856803882","heartbeat_front_addr":"172.21.15.187:6821/2856803882","state":["exists","up"]},{"osd":9,"uuid":"cf417a8e-e576-43de-b89c-f104ee3969ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1392813017},{"type":"v1","addr":"172.21.15.29:6827","nonce":1392813017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1392813017},{"type":"v1","addr":"172.21.15.29:6829","nonce":1392813017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1392813017},{"type":"v1","addr":"172.21.15.29:6833","nonce":1392813017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1392813017},{"type":"v1","addr":"172.21.15.29:6831","nonce":1392813017}]},"public_addr":"172.21.15.29:6827/1392813017","cluster_addr":"172.21.15.29:6829/1392813017","heartbeat_back_addr":"172.21.15.29:6833/1392813017","heartbeat_front_addr":"172.21.15.29:6831/1392813017","state":["exists","up"]},{"osd":10,"uuid":"bc7cfb9d-3bfa-45b2-929a-a8dc90f1a2a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":2523735721},{"type":"v1","addr":"172.21.15.163:6825","nonce":2523735721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":2523735721},{"type":"v1","addr":"172.21.15.163:6827","nonce":2523735721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":2523735721},{"type":"v1","addr":"172.21.15.163:6831","nonce":2523735721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":2523735721},{"type":"v1","addr":"172.21.15.163:6829","nonce":2523735721}]},"public_addr":"172.21.15.163:6825/2523735721","cluster_addr":"172.21.15.163:6827/2523735721","heartbeat_back_addr":"172.21.15.163:6831/2523735721","heartbeat_front_addr":"172.21.15.163:6829/2523735721","state":["exists","up"]},{"osd":11,"uuid":"1453684e-fe11-4cbc-a868-cf90b3a7a021","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1828122314},{"type":"v1","addr":"172.21.15.187:6825","nonce":1828122314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1828122314},{"type":"v1","addr":"172.21.15.187:6827","nonce":1828122314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":1828122314},{"type":"v1","addr":"172.21.15.187:6831","nonce":1828122314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":1828122314},{"type":"v1","addr":"172.21.15.187:6829","nonce":1828122314}]},"public_addr":"172.21.15.187:6825/1828122314","cluster_addr":"172.21.15.187:6827/1828122314","heartbeat_back_addr":"172.21.15.187:6831/1828122314","heartbeat_front_addr":"172.21.15.187:6829/1828122314","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:31:43.572543+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:32:00.759082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:32:17.705002+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:32:35.666788+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:32:53.221701+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:33:10.290988+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:33:27.946743+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:33:45.698074+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:34:02.750761+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:34:20.662278+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:34:38.233389+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:34:55.419540+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/1419474286":"2024-01-25T13:30:30.728638+0000","172.21.15.29:6800/835547951":"2024-01-25T13:30:08.775587+0000","172.21.15.29:6801/1996568496":"2024-01-25T13:30:30.728638+0000","172.21.15.29:0/2727527401":"2024-01-25T13:30:08.775587+0000","172.21.15.29:0/41509684":"2024-01-25T13:30:30.728638+0000","172.21.15.29:6800/1996568496":"2024-01-25T13:30:30.728638+0000","172.21.15.29:0/2833864599":"2024-01-25T13:30:30.728638+0000","172.21.15.29:0/168349482":"2024-01-25T13:30:08.775587+0000","172.21.15.29:6801/835547951":"2024-01-25T13:30:08.775587+0000","172.21.15.29:0/1142561996":"2024-01-25T13:30:08.775587+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T13:35:46.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.848+0000 7faef3e44700 1 -- 172.21.15.29:0/3564894799 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7faed80a0570 msgr2=0x7faed80a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:46.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.848+0000 7faef3e44700 1 --2- 172.21.15.29:0/3564894799 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7faed80a0570 0x7faed80a2a30 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7faeec100c70 tx=0x7faee8009260 comp rx=0 tx=0).stop 2024-01-24T13:35:46.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.848+0000 7faef3e44700 1 -- 172.21.15.29:0/3564894799 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faeec1017b0 msgr2=0x7faeec19e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:46.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.848+0000 7faef3e44700 1 --2- 172.21.15.29:0/3564894799 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faeec1017b0 0x7faeec19e020 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7faee4008f10 tx=0x7faee4008f40 comp rx=0 tx=0).stop 2024-01-24T13:35:46.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.848+0000 7faef3e44700 1 -- 172.21.15.29:0/3564894799 shutdown_connections 2024-01-24T13:35:46.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.848+0000 7faef3e44700 1 --2- 172.21.15.29:0/3564894799 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7faed80a0570 0x7faed80a2a30 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:46.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.848+0000 7faef3e44700 1 --2- 172.21.15.29:0/3564894799 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faeec1017b0 0x7faeec19e020 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:46.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.848+0000 7faef3e44700 1 --2- 172.21.15.29:0/3564894799 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faeec0ffc20 0x7faeec196a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:46.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.848+0000 7faef3e44700 1 --2- 172.21.15.29:0/3564894799 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faeec100e10 0x7faeec196fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:46.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.848+0000 7faef3e44700 1 -- 172.21.15.29:0/3564894799 >> 172.21.15.29:0/3564894799 conn(0x7faeec0fb170 msgr2=0x7faeec067f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:46.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.848+0000 7faef3e44700 1 -- 172.21.15.29:0/3564894799 shutdown_connections 2024-01-24T13:35:46.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:46.849+0000 7faef3e44700 1 -- 172.21.15.29:0/3564894799 wait complete. 2024-01-24T13:35:46.864 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T13:35:47.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.000+0000 7f859ee2e700 1 -- 172.21.15.29:0/472688184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85980ffc60 msgr2=0x7f8598100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:47.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.000+0000 7f859ee2e700 1 --2- 172.21.15.29:0/472688184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85980ffc60 0x7f8598100080 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f858c0099c0 tx=0x7f858c009cd0 comp rx=0 tx=0).stop 2024-01-24T13:35:47.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.000+0000 7f859ee2e700 1 -- 172.21.15.29:0/472688184 shutdown_connections 2024-01-24T13:35:47.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.000+0000 7f859ee2e700 1 --2- 172.21.15.29:0/472688184 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85981017f0 0x7f85981080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:47.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.000+0000 7f859ee2e700 1 --2- 172.21.15.29:0/472688184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85980ffc60 0x7f8598100080 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:47.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.000+0000 7f859ee2e700 1 --2- 172.21.15.29:0/472688184 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8598100e50 0x7f85981012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:47.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.000+0000 7f859ee2e700 1 -- 172.21.15.29:0/472688184 >> 172.21.15.29:0/472688184 conn(0x7f85980fb1d0 msgr2=0x7f85980fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:47.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.000+0000 7f859ee2e700 1 -- 172.21.15.29:0/472688184 shutdown_connections 2024-01-24T13:35:47.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.000+0000 7f859ee2e700 1 -- 172.21.15.29:0/472688184 wait complete. 2024-01-24T13:35:47.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.000+0000 7f859ee2e700 1 Processor -- start 2024-01-24T13:35:47.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.001+0000 7f859ee2e700 1 -- start start 2024-01-24T13:35:47.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.001+0000 7f859ee2e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f85980ffc60 0x7f8598103bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:47.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.001+0000 7f859ee2e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8598100e50 0x7f8598102230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:47.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.001+0000 7f859ee2e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85981017f0 0x7f8598102770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:47.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.001+0000 7f859ee2e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8598102f10 con 0x7f85981017f0 2024-01-24T13:35:47.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.001+0000 7f859ee2e700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8598103080 con 0x7f85980ffc60 2024-01-24T13:35:47.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.001+0000 7f859ee2e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85981a0a70 con 0x7f8598100e50 2024-01-24T13:35:47.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.001+0000 7f859d3cb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85981017f0 0x7f8598102770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:47.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.001+0000 7f859d3cb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85981017f0 0x7f8598102770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44852/0 (socket says 172.21.15.29:44852) 2024-01-24T13:35:47.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.001+0000 7f859d3cb700 1 -- 172.21.15.29:0/1611582776 learned_addr learned my addr 172.21.15.29:0/1611582776 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:47.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.001+0000 7f859cbca700 1 --2- 172.21.15.29:0/1611582776 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f85980ffc60 0x7f8598103bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:47.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.001+0000 7f8597fff700 1 --2- 172.21.15.29:0/1611582776 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8598100e50 0x7f8598102230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:47.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.001+0000 7f859d3cb700 1 -- 172.21.15.29:0/1611582776 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8598100e50 msgr2=0x7f8598102230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:47.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.001+0000 7f859d3cb700 1 --2- 172.21.15.29:0/1611582776 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8598100e50 0x7f8598102230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:47.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.001+0000 7f859d3cb700 1 -- 172.21.15.29:0/1611582776 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f85980ffc60 msgr2=0x7f8598103bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:47.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.001+0000 7f859d3cb700 1 --2- 172.21.15.29:0/1611582776 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f85980ffc60 0x7f8598103bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:47.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.001+0000 7f859d3cb700 1 -- 172.21.15.29:0/1611582776 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f858c0095f0 con 0x7f85981017f0 2024-01-24T13:35:47.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.001+0000 7f859d3cb700 1 --2- 172.21.15.29:0/1611582776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85981017f0 0x7f8598102770 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f859000ef30 tx=0x7f859000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:47.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.002+0000 7f8595ffb700 1 -- 172.21.15.29:0/1611582776 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8590018070 con 0x7f85981017f0 2024-01-24T13:35:47.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.002+0000 7f8595ffb700 1 -- 172.21.15.29:0/1611582776 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f859000f040 con 0x7f85981017f0 2024-01-24T13:35:47.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.002+0000 7f859ee2e700 1 -- 172.21.15.29:0/1611582776 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85981a0cf0 con 0x7f85981017f0 2024-01-24T13:35:47.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.002+0000 7f8595ffb700 1 -- 172.21.15.29:0/1611582776 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859000cdb0 con 0x7f85981017f0 2024-01-24T13:35:47.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.002+0000 7f859ee2e700 1 -- 172.21.15.29:0/1611582776 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85981a11f0 con 0x7f85981017f0 2024-01-24T13:35:47.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.003+0000 7f859ee2e700 1 -- 172.21.15.29:0/1611582776 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f859810b4c0 con 0x7f85981017f0 2024-01-24T13:35:47.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.004+0000 7f8595ffb700 1 -- 172.21.15.29:0/1611582776 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f8590003a00 con 0x7f85981017f0 2024-01-24T13:35:47.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.004+0000 7f8595ffb700 1 --2- 172.21.15.29:0/1611582776 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f85880a05f0 0x7f85880a2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:47.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.005+0000 7f8595ffb700 1 -- 172.21.15.29:0/1611582776 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f85900cb700 con 0x7f85981017f0 2024-01-24T13:35:47.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.005+0000 7f859cbca700 1 --2- 172.21.15.29:0/1611582776 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f85880a05f0 0x7f85880a2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:47.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.005+0000 7f859cbca700 1 --2- 172.21.15.29:0/1611582776 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f85880a05f0 0x7f85880a2ab0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f858c009990 tx=0x7f858c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:47.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.009+0000 7f8595ffb700 1 -- 172.21.15.29:0/1611582776 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f85900950e0 con 0x7f85981017f0 2024-01-24T13:35:47.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.225+0000 7f859ee2e700 1 -- 172.21.15.29:0/1611582776 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f85981a1800 con 0x7f85981017f0 2024-01-24T13:35:47.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.227+0000 7f8595ffb700 1 -- 172.21.15.29:0/1611582776 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23981 (secure 0 0 0) 0x7f8590098700 con 0x7f85981017f0 2024-01-24T13:35:47.228 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-24T13:35:47.228 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":85,"fsid":"83afcf5e-babc-11ee-95b1-87774f69a715","created":"2024-01-24T13:29:33.175895+0000","modified":"2024-01-24T13:35:34.866792+0000","last_up_change":"2024-01-24T13:34:57.141902+0000","last_in_change":"2024-01-24T13:34:41.071943+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T13:32:21.264634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-24T13:35:28.761733+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-24T13:35:29.763293+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-01-24T13:35:32.858844+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9857a607-2e6f-41a5-a86a-795970be6210","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":2406629076},{"type":"v1","addr":"172.21.15.29:6803","nonce":2406629076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":2406629076},{"type":"v1","addr":"172.21.15.29:6805","nonce":2406629076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2406629076},{"type":"v1","addr":"172.21.15.29:6809","nonce":2406629076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":2406629076},{"type":"v1","addr":"172.21.15.29:6807","nonce":2406629076}]},"public_addr":"172.21.15.29:6803/2406629076","cluster_addr":"172.21.15.29:6805/2406629076","heartbeat_back_addr":"172.21.15.29:6809/2406629076","heartbeat_front_addr":"172.21.15.29:6807/2406629076","state":["exists","up"]},{"osd":1,"uuid":"8bb2ba48-23d4-4bbd-a912-ae74f9e7f6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":2217244566},{"type":"v1","addr":"172.21.15.163:6801","nonce":2217244566}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":2217244566},{"type":"v1","addr":"172.21.15.163:6803","nonce":2217244566}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":2217244566},{"type":"v1","addr":"172.21.15.163:6807","nonce":2217244566}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":2217244566},{"type":"v1","addr":"172.21.15.163:6805","nonce":2217244566}]},"public_addr":"172.21.15.163:6801/2217244566","cluster_addr":"172.21.15.163:6803/2217244566","heartbeat_back_addr":"172.21.15.163:6807/2217244566","heartbeat_front_addr":"172.21.15.163:6805/2217244566","state":["exists","up"]},{"osd":2,"uuid":"f0ed4131-cd29-4706-89ea-fde40f46ea83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3589694992},{"type":"v1","addr":"172.21.15.187:6801","nonce":3589694992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3589694992},{"type":"v1","addr":"172.21.15.187:6803","nonce":3589694992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3589694992},{"type":"v1","addr":"172.21.15.187:6807","nonce":3589694992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3589694992},{"type":"v1","addr":"172.21.15.187:6805","nonce":3589694992}]},"public_addr":"172.21.15.187:6801/3589694992","cluster_addr":"172.21.15.187:6803/3589694992","heartbeat_back_addr":"172.21.15.187:6807/3589694992","heartbeat_front_addr":"172.21.15.187:6805/3589694992","state":["exists","up"]},{"osd":3,"uuid":"da821310-b502-4b77-a5ce-0ffed0a3dafc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3417210415},{"type":"v1","addr":"172.21.15.29:6811","nonce":3417210415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":3417210415},{"type":"v1","addr":"172.21.15.29:6813","nonce":3417210415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3417210415},{"type":"v1","addr":"172.21.15.29:6817","nonce":3417210415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3417210415},{"type":"v1","addr":"172.21.15.29:6815","nonce":3417210415}]},"public_addr":"172.21.15.29:6811/3417210415","cluster_addr":"172.21.15.29:6813/3417210415","heartbeat_back_addr":"172.21.15.29:6817/3417210415","heartbeat_front_addr":"172.21.15.29:6815/3417210415","state":["exists","up"]},{"osd":4,"uuid":"8b3a348f-e7dd-4ccb-82a5-2d3f5d101da8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3283787183},{"type":"v1","addr":"172.21.15.163:6809","nonce":3283787183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":3283787183},{"type":"v1","addr":"172.21.15.163:6811","nonce":3283787183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3283787183},{"type":"v1","addr":"172.21.15.163:6815","nonce":3283787183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":3283787183},{"type":"v1","addr":"172.21.15.163:6813","nonce":3283787183}]},"public_addr":"172.21.15.163:6809/3283787183","cluster_addr":"172.21.15.163:6811/3283787183","heartbeat_back_addr":"172.21.15.163:6815/3283787183","heartbeat_front_addr":"172.21.15.163:6813/3283787183","state":["exists","up"]},{"osd":5,"uuid":"c26c7155-618b-4712-acdb-d4fd6c8b7f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":2259228817},{"type":"v1","addr":"172.21.15.187:6809","nonce":2259228817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":2259228817},{"type":"v1","addr":"172.21.15.187:6811","nonce":2259228817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":2259228817},{"type":"v1","addr":"172.21.15.187:6815","nonce":2259228817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":2259228817},{"type":"v1","addr":"172.21.15.187:6813","nonce":2259228817}]},"public_addr":"172.21.15.187:6809/2259228817","cluster_addr":"172.21.15.187:6811/2259228817","heartbeat_back_addr":"172.21.15.187:6815/2259228817","heartbeat_front_addr":"172.21.15.187:6813/2259228817","state":["exists","up"]},{"osd":6,"uuid":"042ac0b4-1fe1-444f-bf51-ad89613a2185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":938400701},{"type":"v1","addr":"172.21.15.29:6819","nonce":938400701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":938400701},{"type":"v1","addr":"172.21.15.29:6821","nonce":938400701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":938400701},{"type":"v1","addr":"172.21.15.29:6825","nonce":938400701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":938400701},{"type":"v1","addr":"172.21.15.29:6823","nonce":938400701}]},"public_addr":"172.21.15.29:6819/938400701","cluster_addr":"172.21.15.29:6821/938400701","heartbeat_back_addr":"172.21.15.29:6825/938400701","heartbeat_front_addr":"172.21.15.29:6823/938400701","state":["exists","up"]},{"osd":7,"uuid":"add3d09b-76f3-4a93-83a2-30ac94c3fad1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2783976092},{"type":"v1","addr":"172.21.15.163:6817","nonce":2783976092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2783976092},{"type":"v1","addr":"172.21.15.163:6819","nonce":2783976092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2783976092},{"type":"v1","addr":"172.21.15.163:6823","nonce":2783976092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2783976092},{"type":"v1","addr":"172.21.15.163:6821","nonce":2783976092}]},"public_addr":"172.21.15.163:6817/2783976092","cluster_addr":"172.21.15.163:6819/2783976092","heartbeat_back_addr":"172.21.15.163:6823/2783976092","heartbeat_front_addr":"172.21.15.163:6821/2783976092","state":["exists","up"]},{"osd":8,"uuid":"71044d6a-b55f-4bcf-94c5-79f71eb7b422","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":2856803882},{"type":"v1","addr":"172.21.15.187:6817","nonce":2856803882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":2856803882},{"type":"v1","addr":"172.21.15.187:6819","nonce":2856803882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":2856803882},{"type":"v1","addr":"172.21.15.187:6823","nonce":2856803882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":2856803882},{"type":"v1","addr":"172.21.15.187:6821","nonce":2856803882}]},"public_addr":"172.21.15.187:6817/2856803882","cluster_addr":"172.21.15.187:6819/2856803882","heartbeat_back_addr":"172.21.15.187:6823/2856803882","heartbeat_front_addr":"172.21.15.187:6821/2856803882","state":["exists","up"]},{"osd":9,"uuid":"cf417a8e-e576-43de-b89c-f104ee3969ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1392813017},{"type":"v1","addr":"172.21.15.29:6827","nonce":1392813017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1392813017},{"type":"v1","addr":"172.21.15.29:6829","nonce":1392813017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1392813017},{"type":"v1","addr":"172.21.15.29:6833","nonce":1392813017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1392813017},{"type":"v1","addr":"172.21.15.29:6831","nonce":1392813017}]},"public_addr":"172.21.15.29:6827/1392813017","cluster_addr":"172.21.15.29:6829/1392813017","heartbeat_back_addr":"172.21.15.29:6833/1392813017","heartbeat_front_addr":"172.21.15.29:6831/1392813017","state":["exists","up"]},{"osd":10,"uuid":"bc7cfb9d-3bfa-45b2-929a-a8dc90f1a2a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":2523735721},{"type":"v1","addr":"172.21.15.163:6825","nonce":2523735721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":2523735721},{"type":"v1","addr":"172.21.15.163:6827","nonce":2523735721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":2523735721},{"type":"v1","addr":"172.21.15.163:6831","nonce":2523735721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":2523735721},{"type":"v1","addr":"172.21.15.163:6829","nonce":2523735721}]},"public_addr":"172.21.15.163:6825/2523735721","cluster_addr":"172.21.15.163:6827/2523735721","heartbeat_back_addr":"172.21.15.163:6831/2523735721","heartbeat_front_addr":"172.21.15.163:6829/2523735721","state":["exists","up"]},{"osd":11,"uuid":"1453684e-fe11-4cbc-a868-cf90b3a7a021","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1828122314},{"type":"v1","addr":"172.21.15.187:6825","nonce":1828122314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1828122314},{"type":"v1","addr":"172.21.15.187:6827","nonce":1828122314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":1828122314},{"type":"v1","addr":"172.21.15.187:6831","nonce":1828122314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":1828122314},{"type":"v1","addr":"172.21.15.187:6829","nonce":1828122314}]},"public_addr":"172.21.15.187:6825/1828122314","cluster_addr":"172.21.15.187:6827/1828122314","heartbeat_back_addr":"172.21.15.187:6831/1828122314","heartbeat_front_addr":"172.21.15.187:6829/1828122314","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:31:43.572543+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:32:00.759082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:32:17.705002+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:32:35.666788+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:32:53.221701+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:33:10.290988+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:33:27.946743+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:33:45.698074+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:34:02.750761+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:34:20.662278+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:34:38.233389+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T13:34:55.419540+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/1419474286":"2024-01-25T13:30:30.728638+0000","172.21.15.29:6800/835547951":"2024-01-25T13:30:08.775587+0000","172.21.15.29:6801/1996568496":"2024-01-25T13:30:30.728638+0000","172.21.15.29:0/2727527401":"2024-01-25T13:30:08.775587+0000","172.21.15.29:0/41509684":"2024-01-25T13:30:30.728638+0000","172.21.15.29:6800/1996568496":"2024-01-25T13:30:30.728638+0000","172.21.15.29:0/2833864599":"2024-01-25T13:30:30.728638+0000","172.21.15.29:0/168349482":"2024-01-25T13:30:08.775587+0000","172.21.15.29:6801/835547951":"2024-01-25T13:30:08.775587+0000","172.21.15.29:0/1142561996":"2024-01-25T13:30:08.775587+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T13:35:47.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.229+0000 7f859ee2e700 1 -- 172.21.15.29:0/1611582776 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f85880a05f0 msgr2=0x7f85880a2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:47.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.229+0000 7f859ee2e700 1 --2- 172.21.15.29:0/1611582776 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f85880a05f0 0x7f85880a2ab0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f858c009990 tx=0x7f858c005fb0 comp rx=0 tx=0).stop 2024-01-24T13:35:47.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.229+0000 7f859ee2e700 1 -- 172.21.15.29:0/1611582776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85981017f0 msgr2=0x7f8598102770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:47.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.229+0000 7f859ee2e700 1 --2- 172.21.15.29:0/1611582776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85981017f0 0x7f8598102770 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f859000ef30 tx=0x7f859000c3a0 comp rx=0 tx=0).stop 2024-01-24T13:35:47.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.229+0000 7f859ee2e700 1 -- 172.21.15.29:0/1611582776 shutdown_connections 2024-01-24T13:35:47.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.229+0000 7f859ee2e700 1 --2- 172.21.15.29:0/1611582776 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f85880a05f0 0x7f85880a2ab0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:47.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.229+0000 7f859ee2e700 1 --2- 172.21.15.29:0/1611582776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85981017f0 0x7f8598102770 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:47.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.229+0000 7f859ee2e700 1 --2- 172.21.15.29:0/1611582776 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f85980ffc60 0x7f8598103bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:47.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.229+0000 7f859ee2e700 1 --2- 172.21.15.29:0/1611582776 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8598100e50 0x7f8598102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:47.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.229+0000 7f859ee2e700 1 -- 172.21.15.29:0/1611582776 >> 172.21.15.29:0/1611582776 conn(0x7f85980fb1d0 msgr2=0x7f8598068f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:47.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.229+0000 7f859ee2e700 1 -- 172.21.15.29:0/1611582776 shutdown_connections 2024-01-24T13:35:47.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.229+0000 7f859ee2e700 1 -- 172.21.15.29:0/1611582776 wait complete. 2024-01-24T13:35:47.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:46 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/823435214' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-01-24T13:35:47.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:46 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/1465309573' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-01-24T13:35:47.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:46 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/3564894799' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T13:35:47.243 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:46 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/823435214' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-01-24T13:35:47.243 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:46 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/1465309573' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-01-24T13:35:47.243 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:46 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/3564894799' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T13:35:47.244 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-24T13:35:47.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:46 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/823435214' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-01-24T13:35:47.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:46 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/1465309573' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-01-24T13:35:47.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:46 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/3564894799' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T13:35:47.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.384+0000 7fb0687b5700 1 -- 172.21.15.29:0/1578853957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0601017c0 msgr2=0x7fb060108070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:47.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.384+0000 7fb0687b5700 1 --2- 172.21.15.29:0/1578853957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0601017c0 0x7fb060108070 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb05800b0d0 tx=0x7fb05800b3e0 comp rx=0 tx=0).stop 2024-01-24T13:35:47.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.384+0000 7fb0687b5700 1 -- 172.21.15.29:0/1578853957 shutdown_connections 2024-01-24T13:35:47.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.384+0000 7fb0687b5700 1 --2- 172.21.15.29:0/1578853957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0601017c0 0x7fb060108070 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:47.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.384+0000 7fb0687b5700 1 --2- 172.21.15.29:0/1578853957 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0600ffc30 0x7fb060100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:47.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.384+0000 7fb0687b5700 1 --2- 172.21.15.29:0/1578853957 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb060100e20 0x7fb060101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:47.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.384+0000 7fb0687b5700 1 -- 172.21.15.29:0/1578853957 >> 172.21.15.29:0/1578853957 conn(0x7fb0600fb1a0 msgr2=0x7fb0600fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:47.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.384+0000 7fb0687b5700 1 -- 172.21.15.29:0/1578853957 shutdown_connections 2024-01-24T13:35:47.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.384+0000 7fb0687b5700 1 -- 172.21.15.29:0/1578853957 wait complete. 2024-01-24T13:35:47.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.385+0000 7fb0687b5700 1 Processor -- start 2024-01-24T13:35:47.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.385+0000 7fb0687b5700 1 -- start start 2024-01-24T13:35:47.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.385+0000 7fb0687b5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0600ffc30 0x7fb060196b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:47.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.385+0000 7fb0687b5700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb060100e20 0x7fb060197050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:47.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.385+0000 7fb0687b5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0601017c0 0x7fb06019e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:47.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.385+0000 7fb0687b5700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06019e690 con 0x7fb0601017c0 2024-01-24T13:35:47.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.385+0000 7fb0687b5700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06019e7d0 con 0x7fb060100e20 2024-01-24T13:35:47.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.385+0000 7fb0687b5700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06019e960 con 0x7fb0600ffc30 2024-01-24T13:35:47.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.385+0000 7fb066d52700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0601017c0 0x7fb06019e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:47.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.385+0000 7fb066d52700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0601017c0 0x7fb06019e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44864/0 (socket says 172.21.15.29:44864) 2024-01-24T13:35:47.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.385+0000 7fb066d52700 1 -- 172.21.15.29:0/2934917580 learned_addr learned my addr 172.21.15.29:0/2934917580 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:47.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.385+0000 7fb066551700 1 --2- 172.21.15.29:0/2934917580 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0600ffc30 0x7fb060196b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:47.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.385+0000 7fb065d50700 1 --2- 172.21.15.29:0/2934917580 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb060100e20 0x7fb060197050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:47.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.385+0000 7fb066d52700 1 -- 172.21.15.29:0/2934917580 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0600ffc30 msgr2=0x7fb060196b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:47.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.385+0000 7fb066d52700 1 --2- 172.21.15.29:0/2934917580 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0600ffc30 0x7fb060196b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:47.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.385+0000 7fb066d52700 1 -- 172.21.15.29:0/2934917580 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb060100e20 msgr2=0x7fb060197050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:47.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.385+0000 7fb066d52700 1 --2- 172.21.15.29:0/2934917580 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb060100e20 0x7fb060197050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:47.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.385+0000 7fb066d52700 1 -- 172.21.15.29:0/2934917580 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb058009d00 con 0x7fb0601017c0 2024-01-24T13:35:47.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.386+0000 7fb066d52700 1 --2- 172.21.15.29:0/2934917580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0601017c0 0x7fb06019e0c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb058007a00 tx=0x7fb0580094c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:47.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.386+0000 7fb0537fe700 1 -- 172.21.15.29:0/2934917580 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05800e040 con 0x7fb0601017c0 2024-01-24T13:35:47.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.386+0000 7fb0537fe700 1 -- 172.21.15.29:0/2934917580 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb058003e10 con 0x7fb0601017c0 2024-01-24T13:35:47.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.386+0000 7fb0687b5700 1 -- 172.21.15.29:0/2934917580 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb06019ebe0 con 0x7fb0601017c0 2024-01-24T13:35:47.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.386+0000 7fb0537fe700 1 -- 172.21.15.29:0/2934917580 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb058012a90 con 0x7fb0601017c0 2024-01-24T13:35:47.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.386+0000 7fb0687b5700 1 -- 172.21.15.29:0/2934917580 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb06019f060 con 0x7fb0601017c0 2024-01-24T13:35:47.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.387+0000 7fb0687b5700 1 -- 172.21.15.29:0/2934917580 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb06010b450 con 0x7fb0601017c0 2024-01-24T13:35:47.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.388+0000 7fb0537fe700 1 -- 172.21.15.29:0/2934917580 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7fb0580056c0 con 0x7fb0601017c0 2024-01-24T13:35:47.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.389+0000 7fb0537fe700 1 --2- 172.21.15.29:0/2934917580 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb04c0a05a0 0x7fb04c0a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:47.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.389+0000 7fb066551700 1 --2- 172.21.15.29:0/2934917580 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb04c0a05a0 0x7fb04c0a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:47.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.389+0000 7fb0537fe700 1 -- 172.21.15.29:0/2934917580 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fb0580cd950 con 0x7fb0601017c0 2024-01-24T13:35:47.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.390+0000 7fb066551700 1 --2- 172.21.15.29:0/2934917580 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb04c0a05a0 0x7fb04c0a2a60 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb060100c80 tx=0x7fb05c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:47.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.393+0000 7fb0537fe700 1 -- 172.21.15.29:0/2934917580 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fb058097390 con 0x7fb0601017c0 2024-01-24T13:35:47.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.655+0000 7fb0687b5700 1 -- 172.21.15.29:0/2934917580 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fb06019f610 con 0x7fb0601017c0 2024-01-24T13:35:47.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.868+0000 7fb0537fe700 1 -- 172.21.15.29:0/2934917580 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb05809a9b0 con 0x7fb0601017c0 2024-01-24T13:35:47.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.871+0000 7fb0687b5700 1 -- 172.21.15.29:0/2934917580 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb04c0a05a0 msgr2=0x7fb04c0a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:47.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.871+0000 7fb0687b5700 1 --2- 172.21.15.29:0/2934917580 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb04c0a05a0 0x7fb04c0a2a60 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb060100c80 tx=0x7fb05c00b410 comp rx=0 tx=0).stop 2024-01-24T13:35:47.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.871+0000 7fb0687b5700 1 -- 172.21.15.29:0/2934917580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0601017c0 msgr2=0x7fb06019e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:47.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.871+0000 7fb0687b5700 1 --2- 172.21.15.29:0/2934917580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0601017c0 0x7fb06019e0c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb058007a00 tx=0x7fb0580094c0 comp rx=0 tx=0).stop 2024-01-24T13:35:47.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.871+0000 7fb0687b5700 1 -- 172.21.15.29:0/2934917580 shutdown_connections 2024-01-24T13:35:47.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.871+0000 7fb0687b5700 1 --2- 172.21.15.29:0/2934917580 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fb04c0a05a0 0x7fb04c0a2a60 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:47.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.871+0000 7fb0687b5700 1 --2- 172.21.15.29:0/2934917580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0601017c0 0x7fb06019e0c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:47.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.871+0000 7fb0687b5700 1 --2- 172.21.15.29:0/2934917580 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0600ffc30 0x7fb060196b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:47.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.871+0000 7fb0687b5700 1 --2- 172.21.15.29:0/2934917580 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb060100e20 0x7fb060197050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:47.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.871+0000 7fb0687b5700 1 -- 172.21.15.29:0/2934917580 >> 172.21.15.29:0/2934917580 conn(0x7fb0600fb1a0 msgr2=0x7fb060067ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:47.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.871+0000 7fb0687b5700 1 -- 172.21.15.29:0/2934917580 shutdown_connections 2024-01-24T13:35:47.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:47.871+0000 7fb0687b5700 1 -- 172.21.15.29:0/2934917580 wait complete. 2024-01-24T13:35:47.888 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-01-24T13:35:47.888 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-01-24T13:35:48.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.031+0000 7f30307f6700 1 -- 172.21.15.29:0/738051705 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3028101f20 msgr2=0x7f3028102380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:48.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.031+0000 7f30307f6700 1 --2- 172.21.15.29:0/738051705 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3028101f20 0x7f3028102380 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f30180099c0 tx=0x7f3018009cd0 comp rx=0 tx=0).stop 2024-01-24T13:35:48.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.031+0000 7f30307f6700 1 -- 172.21.15.29:0/738051705 shutdown_connections 2024-01-24T13:35:48.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.031+0000 7f30307f6700 1 --2- 172.21.15.29:0/738051705 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f30281028c0 0x7f3028109170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:48.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.031+0000 7f30307f6700 1 --2- 172.21.15.29:0/738051705 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3028100d30 0x7f3028101150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:48.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.031+0000 7f30307f6700 1 --2- 172.21.15.29:0/738051705 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3028101f20 0x7f3028102380 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:48.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.031+0000 7f30307f6700 1 -- 172.21.15.29:0/738051705 >> 172.21.15.29:0/738051705 conn(0x7f30280fc2c0 msgr2=0x7f30280fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:48.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.031+0000 7f30307f6700 1 -- 172.21.15.29:0/738051705 shutdown_connections 2024-01-24T13:35:48.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.031+0000 7f30307f6700 1 -- 172.21.15.29:0/738051705 wait complete. 2024-01-24T13:35:48.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.032+0000 7f30307f6700 1 Processor -- start 2024-01-24T13:35:48.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.032+0000 7f30307f6700 1 -- start start 2024-01-24T13:35:48.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.032+0000 7f30307f6700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3028100d30 0x7f3028196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:48.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.032+0000 7f30307f6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3028101f20 0x7f3028196f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:48.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.032+0000 7f30307f6700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30281028c0 0x7f302819dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:48.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.032+0000 7f30307f6700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302819e580 con 0x7f30281028c0 2024-01-24T13:35:48.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.032+0000 7f30307f6700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302819e6c0 con 0x7f3028100d30 2024-01-24T13:35:48.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.032+0000 7f30307f6700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302819e850 con 0x7f3028101f20 2024-01-24T13:35:48.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.032+0000 7f302ed93700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30281028c0 0x7f302819dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:48.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.032+0000 7f302ed93700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30281028c0 0x7f302819dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44880/0 (socket says 172.21.15.29:44880) 2024-01-24T13:35:48.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.032+0000 7f302ed93700 1 -- 172.21.15.29:0/137184183 learned_addr learned my addr 172.21.15.29:0/137184183 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:48.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.032+0000 7f302e592700 1 --2- 172.21.15.29:0/137184183 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3028100d30 0x7f3028196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:48.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.033+0000 7f302dd91700 1 --2- 172.21.15.29:0/137184183 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3028101f20 0x7f3028196f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:48.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.033+0000 7f302ed93700 1 -- 172.21.15.29:0/137184183 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3028101f20 msgr2=0x7f3028196f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:48.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.033+0000 7f302ed93700 1 --2- 172.21.15.29:0/137184183 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3028101f20 0x7f3028196f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:48.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.033+0000 7f302ed93700 1 -- 172.21.15.29:0/137184183 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3028100d30 msgr2=0x7f3028196a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:48.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.033+0000 7f302ed93700 1 --2- 172.21.15.29:0/137184183 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3028100d30 0x7f3028196a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:48.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.033+0000 7f302ed93700 1 -- 172.21.15.29:0/137184183 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30180095f0 con 0x7f30281028c0 2024-01-24T13:35:48.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.033+0000 7f302ed93700 1 --2- 172.21.15.29:0/137184183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30281028c0 0x7f302819dfb0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3020007ee0 tx=0x7f302000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:48.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.033+0000 7f301f7fe700 1 -- 172.21.15.29:0/137184183 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f302000dd20 con 0x7f30281028c0 2024-01-24T13:35:48.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.033+0000 7f301f7fe700 1 -- 172.21.15.29:0/137184183 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f302000f040 con 0x7f30281028c0 2024-01-24T13:35:48.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.033+0000 7f30307f6700 1 -- 172.21.15.29:0/137184183 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f302819eb30 con 0x7f30281028c0 2024-01-24T13:35:48.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.033+0000 7f301f7fe700 1 -- 172.21.15.29:0/137184183 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30200048a0 con 0x7f30281028c0 2024-01-24T13:35:48.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.033+0000 7f30307f6700 1 -- 172.21.15.29:0/137184183 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f302819f040 con 0x7f30281028c0 2024-01-24T13:35:48.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.034+0000 7f30307f6700 1 -- 172.21.15.29:0/137184183 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3028107cb0 con 0x7f30281028c0 2024-01-24T13:35:48.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.035+0000 7f301f7fe700 1 -- 172.21.15.29:0/137184183 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f3020003a00 con 0x7f30281028c0 2024-01-24T13:35:48.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.036+0000 7f301f7fe700 1 --2- 172.21.15.29:0/137184183 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f30140a05a0 0x7f30140a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:48.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.036+0000 7f301f7fe700 1 -- 172.21.15.29:0/137184183 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f30200cbc10 con 0x7f30281028c0 2024-01-24T13:35:48.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.036+0000 7f302e592700 1 --2- 172.21.15.29:0/137184183 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f30140a05a0 0x7f30140a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:48.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.037+0000 7f302e592700 1 --2- 172.21.15.29:0/137184183 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f30140a05a0 0x7f30140a2a60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3028101d80 tx=0x7f302400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:48.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.040+0000 7f301f7fe700 1 -- 172.21.15.29:0/137184183 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f3020095560 con 0x7f30281028c0 2024-01-24T13:35:48.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:47 smithi163 ceph-mon[44969]: pgmap v233: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 944 B/s wr, 2 op/s 2024-01-24T13:35:48.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:47 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/1611582776' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T13:35:48.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:47 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/2934917580' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-01-24T13:35:48.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.302+0000 7f30307f6700 1 -- 172.21.15.29:0/137184183 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7f302802ce00 con 0x7f30281028c0 2024-01-24T13:35:48.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:47 smithi187 ceph-mon[44911]: pgmap v233: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 944 B/s wr, 2 op/s 2024-01-24T13:35:48.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:47 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/1611582776' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T13:35:48.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:47 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/2934917580' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-01-24T13:35:48.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:47 smithi029 ceph-mon[40651]: pgmap v233: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 944 B/s wr, 2 op/s 2024-01-24T13:35:48.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:47 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/1611582776' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T13:35:48.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:47 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/2934917580' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-01-24T13:35:48.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.879+0000 7f301f7fe700 1 -- 172.21.15.29:0/137184183 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v18) v1 ==== 105+0+0 (secure 0 0 0) 0x7f3020023020 con 0x7f30281028c0 2024-01-24T13:35:48.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.883+0000 7f30307f6700 1 -- 172.21.15.29:0/137184183 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f30140a05a0 msgr2=0x7f30140a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:48.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.883+0000 7f30307f6700 1 --2- 172.21.15.29:0/137184183 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f30140a05a0 0x7f30140a2a60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3028101d80 tx=0x7f302400b410 comp rx=0 tx=0).stop 2024-01-24T13:35:48.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.883+0000 7f30307f6700 1 -- 172.21.15.29:0/137184183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30281028c0 msgr2=0x7f302819dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:48.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.883+0000 7f30307f6700 1 --2- 172.21.15.29:0/137184183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30281028c0 0x7f302819dfb0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3020007ee0 tx=0x7f302000d3b0 comp rx=0 tx=0).stop 2024-01-24T13:35:48.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.883+0000 7f30307f6700 1 -- 172.21.15.29:0/137184183 shutdown_connections 2024-01-24T13:35:48.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.884+0000 7f30307f6700 1 --2- 172.21.15.29:0/137184183 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f30140a05a0 0x7f30140a2a60 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:48.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.884+0000 7f30307f6700 1 --2- 172.21.15.29:0/137184183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30281028c0 0x7f302819dfb0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:48.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.884+0000 7f30307f6700 1 --2- 172.21.15.29:0/137184183 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3028100d30 0x7f3028196a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:48.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.884+0000 7f30307f6700 1 --2- 172.21.15.29:0/137184183 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3028101f20 0x7f3028196f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:48.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.884+0000 7f30307f6700 1 -- 172.21.15.29:0/137184183 >> 172.21.15.29:0/137184183 conn(0x7f30280fc2c0 msgr2=0x7f30280fe560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:48.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.884+0000 7f30307f6700 1 -- 172.21.15.29:0/137184183 shutdown_connections 2024-01-24T13:35:48.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:48.884+0000 7f30307f6700 1 -- 172.21.15.29:0/137184183 wait complete. 2024-01-24T13:35:48.903 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-01-24T13:35:49.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.047+0000 7fac9962b700 1 -- 172.21.15.29:0/3195326010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac94100da0 msgr2=0x7fac941011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:49.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.047+0000 7fac9962b700 1 --2- 172.21.15.29:0/3195326010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac94100da0 0x7fac941011c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fac840099c0 tx=0x7fac84009cd0 comp rx=0 tx=0).stop 2024-01-24T13:35:49.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.047+0000 7fac9962b700 1 -- 172.21.15.29:0/3195326010 shutdown_connections 2024-01-24T13:35:49.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.047+0000 7fac9962b700 1 --2- 172.21.15.29:0/3195326010 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac94102930 0x7fac941091e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:49.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.047+0000 7fac9962b700 1 --2- 172.21.15.29:0/3195326010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac94100da0 0x7fac941011c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:49.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.047+0000 7fac9962b700 1 --2- 172.21.15.29:0/3195326010 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fac94101f90 0x7fac941023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:49.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.047+0000 7fac9962b700 1 -- 172.21.15.29:0/3195326010 >> 172.21.15.29:0/3195326010 conn(0x7fac940fc310 msgr2=0x7fac940fe750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:49.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.047+0000 7fac9962b700 1 -- 172.21.15.29:0/3195326010 shutdown_connections 2024-01-24T13:35:49.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.047+0000 7fac9962b700 1 -- 172.21.15.29:0/3195326010 wait complete. 2024-01-24T13:35:49.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.048+0000 7fac9962b700 1 Processor -- start 2024-01-24T13:35:49.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.048+0000 7fac9962b700 1 -- start start 2024-01-24T13:35:49.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.048+0000 7fac9962b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac94100da0 0x7fac94196a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:49.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.048+0000 7fac9962b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fac94101f90 0x7fac94196fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:49.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.048+0000 7fac9962b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac94102930 0x7fac9419e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:49.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.048+0000 7fac9962b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9419e600 con 0x7fac94102930 2024-01-24T13:35:49.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.048+0000 7fac9962b700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9419e740 con 0x7fac94101f90 2024-01-24T13:35:49.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.048+0000 7fac9962b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9419e8d0 con 0x7fac94100da0 2024-01-24T13:35:49.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.048+0000 7fac937fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac94102930 0x7fac9419e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:49.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.048+0000 7fac937fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac94102930 0x7fac9419e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44892/0 (socket says 172.21.15.29:44892) 2024-01-24T13:35:49.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.048+0000 7fac937fe700 1 -- 172.21.15.29:0/2083360077 learned_addr learned my addr 172.21.15.29:0/2083360077 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:49.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.048+0000 7fac92ffd700 1 --2- 172.21.15.29:0/2083360077 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac94100da0 0x7fac94196a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:49.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.048+0000 7fac927fc700 1 --2- 172.21.15.29:0/2083360077 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fac94101f90 0x7fac94196fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:49.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.048+0000 7fac937fe700 1 -- 172.21.15.29:0/2083360077 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac94100da0 msgr2=0x7fac94196a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:49.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.048+0000 7fac937fe700 1 --2- 172.21.15.29:0/2083360077 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac94100da0 0x7fac94196a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:49.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.048+0000 7fac937fe700 1 -- 172.21.15.29:0/2083360077 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fac94101f90 msgr2=0x7fac94196fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:49.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.049+0000 7fac937fe700 1 --2- 172.21.15.29:0/2083360077 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fac94101f90 0x7fac94196fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:49.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.049+0000 7fac937fe700 1 -- 172.21.15.29:0/2083360077 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac840095f0 con 0x7fac94102930 2024-01-24T13:35:49.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.049+0000 7fac927fc700 1 --2- 172.21.15.29:0/2083360077 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fac94101f90 0x7fac94196fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T13:35:49.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.049+0000 7fac937fe700 1 --2- 172.21.15.29:0/2083360077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac94102930 0x7fac9419e030 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fac8c007ee0 tx=0x7fac8c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:49.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.049+0000 7fac7bfff700 1 -- 172.21.15.29:0/2083360077 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8c00ddd0 con 0x7fac94102930 2024-01-24T13:35:49.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.049+0000 7fac7bfff700 1 -- 172.21.15.29:0/2083360077 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fac8c00f040 con 0x7fac94102930 2024-01-24T13:35:49.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.049+0000 7fac9962b700 1 -- 172.21.15.29:0/2083360077 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac9419ebb0 con 0x7fac94102930 2024-01-24T13:35:49.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.049+0000 7fac7bfff700 1 -- 172.21.15.29:0/2083360077 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8c0048e0 con 0x7fac94102930 2024-01-24T13:35:49.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.049+0000 7fac9962b700 1 -- 172.21.15.29:0/2083360077 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac9419f0c0 con 0x7fac94102930 2024-01-24T13:35:49.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.050+0000 7fac9962b700 1 -- 172.21.15.29:0/2083360077 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac9410c590 con 0x7fac94102930 2024-01-24T13:35:49.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.051+0000 7fac7bfff700 1 -- 172.21.15.29:0/2083360077 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7fac8c003a00 con 0x7fac94102930 2024-01-24T13:35:49.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.052+0000 7fac7bfff700 1 --2- 172.21.15.29:0/2083360077 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fac7c0a05a0 0x7fac7c0a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:49.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.052+0000 7fac7bfff700 1 -- 172.21.15.29:0/2083360077 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fac8c0cb460 con 0x7fac94102930 2024-01-24T13:35:49.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.052+0000 7fac92ffd700 1 --2- 172.21.15.29:0/2083360077 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fac7c0a05a0 0x7fac7c0a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:49.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.052+0000 7fac92ffd700 1 --2- 172.21.15.29:0/2083360077 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fac7c0a05a0 0x7fac7c0a2a60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fac84009f50 tx=0x7fac840051e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:49.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.056+0000 7fac7bfff700 1 -- 172.21.15.29:0/2083360077 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fac8c094db0 con 0x7fac94102930 2024-01-24T13:35:49.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:48 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/2934917580' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-01-24T13:35:49.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:48 smithi163 ceph-mon[44969]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-01-24T13:35:49.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:48 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/137184183' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-01-24T13:35:49.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:48 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/2934917580' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-01-24T13:35:49.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:48 smithi187 ceph-mon[44911]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-01-24T13:35:49.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:48 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/137184183' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-01-24T13:35:49.311 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:48 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/2934917580' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-01-24T13:35:49.311 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:48 smithi029 ceph-mon[40651]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-01-24T13:35:49.311 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:48 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/137184183' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-01-24T13:35:49.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.325+0000 7fac9962b700 1 -- 172.21.15.29:0/2083360077 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7fac94107cf0 con 0x7fac94102930 2024-01-24T13:35:49.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.885+0000 7fac7bfff700 1 -- 172.21.15.29:0/2083360077 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) v1 ==== 121+0+0 (secure 0 0 0) 0x7fac8c0983d0 con 0x7fac94102930 2024-01-24T13:35:49.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.889+0000 7fac9962b700 1 -- 172.21.15.29:0/2083360077 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fac7c0a05a0 msgr2=0x7fac7c0a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:49.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.889+0000 7fac9962b700 1 --2- 172.21.15.29:0/2083360077 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fac7c0a05a0 0x7fac7c0a2a60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fac84009f50 tx=0x7fac840051e0 comp rx=0 tx=0).stop 2024-01-24T13:35:49.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.889+0000 7fac9962b700 1 -- 172.21.15.29:0/2083360077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac94102930 msgr2=0x7fac9419e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:49.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.889+0000 7fac9962b700 1 --2- 172.21.15.29:0/2083360077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac94102930 0x7fac9419e030 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fac8c007ee0 tx=0x7fac8c00d3b0 comp rx=0 tx=0).stop 2024-01-24T13:35:49.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.889+0000 7fac9962b700 1 -- 172.21.15.29:0/2083360077 shutdown_connections 2024-01-24T13:35:49.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.889+0000 7fac9962b700 1 --2- 172.21.15.29:0/2083360077 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7fac7c0a05a0 0x7fac7c0a2a60 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:49.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.889+0000 7fac9962b700 1 --2- 172.21.15.29:0/2083360077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac94102930 0x7fac9419e030 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:49.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.889+0000 7fac9962b700 1 --2- 172.21.15.29:0/2083360077 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac94100da0 0x7fac94196a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:49.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.889+0000 7fac9962b700 1 --2- 172.21.15.29:0/2083360077 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fac94101f90 0x7fac94196fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:49.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.889+0000 7fac9962b700 1 -- 172.21.15.29:0/2083360077 >> 172.21.15.29:0/2083360077 conn(0x7fac940fc310 msgr2=0x7fac940fe560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:49.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.889+0000 7fac9962b700 1 -- 172.21.15.29:0/2083360077 shutdown_connections 2024-01-24T13:35:49.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:49.889+0000 7fac9962b700 1 -- 172.21.15.29:0/2083360077 wait complete. 2024-01-24T13:35:49.909 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-01-24T13:35:50.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.052+0000 7f565d1c7700 1 -- 172.21.15.29:0/2525156229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56581017c0 msgr2=0x7f5658108070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:50.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.052+0000 7f565d1c7700 1 --2- 172.21.15.29:0/2525156229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56581017c0 0x7f5658108070 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f565000b0d0 tx=0x7f565000b3e0 comp rx=0 tx=0).stop 2024-01-24T13:35:50.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.052+0000 7f565d1c7700 1 -- 172.21.15.29:0/2525156229 shutdown_connections 2024-01-24T13:35:50.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.052+0000 7f565d1c7700 1 --2- 172.21.15.29:0/2525156229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56581017c0 0x7f5658108070 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:50.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.052+0000 7f565d1c7700 1 --2- 172.21.15.29:0/2525156229 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f56580ffc30 0x7f5658100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:50.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.052+0000 7f565d1c7700 1 --2- 172.21.15.29:0/2525156229 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5658100e20 0x7f5658101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:50.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.052+0000 7f565d1c7700 1 -- 172.21.15.29:0/2525156229 >> 172.21.15.29:0/2525156229 conn(0x7f56580fb1a0 msgr2=0x7f56580fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:50.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.052+0000 7f565d1c7700 1 -- 172.21.15.29:0/2525156229 shutdown_connections 2024-01-24T13:35:50.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.052+0000 7f565d1c7700 1 -- 172.21.15.29:0/2525156229 wait complete. 2024-01-24T13:35:50.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.053+0000 7f565d1c7700 1 Processor -- start 2024-01-24T13:35:50.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.054+0000 7f565d1c7700 1 -- start start 2024-01-24T13:35:50.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.054+0000 7f565d1c7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f56580ffc30 0x7f5658196ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:50.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.054+0000 7f565d1c7700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5658100e20 0x7f5658197010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:50.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.054+0000 7f565d1c7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56581017c0 0x7f565819e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:50.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.054+0000 7f565d1c7700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565819e650 con 0x7f56581017c0 2024-01-24T13:35:50.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.054+0000 7f565d1c7700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565819e790 con 0x7f5658100e20 2024-01-24T13:35:50.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.054+0000 7f565d1c7700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565819e920 con 0x7f56580ffc30 2024-01-24T13:35:50.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.054+0000 7f565759e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56581017c0 0x7f565819e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:50.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.054+0000 7f565759e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56581017c0 0x7f565819e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44916/0 (socket says 172.21.15.29:44916) 2024-01-24T13:35:50.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.054+0000 7f565759e700 1 -- 172.21.15.29:0/639995951 learned_addr learned my addr 172.21.15.29:0/639995951 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-01-24T13:35:50.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.054+0000 7f5656d9d700 1 --2- 172.21.15.29:0/639995951 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f56580ffc30 0x7f5658196ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:50.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.054+0000 7f565659c700 1 --2- 172.21.15.29:0/639995951 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5658100e20 0x7f5658197010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:50.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.054+0000 7f565759e700 1 -- 172.21.15.29:0/639995951 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f56580ffc30 msgr2=0x7f5658196ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:50.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.054+0000 7f565759e700 1 --2- 172.21.15.29:0/639995951 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f56580ffc30 0x7f5658196ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:50.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.054+0000 7f565759e700 1 -- 172.21.15.29:0/639995951 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5658100e20 msgr2=0x7f5658197010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:50.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.054+0000 7f565759e700 1 --2- 172.21.15.29:0/639995951 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5658100e20 0x7f5658197010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:50.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.054+0000 7f565759e700 1 -- 172.21.15.29:0/639995951 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5650009d00 con 0x7f56581017c0 2024-01-24T13:35:50.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.054+0000 7f565759e700 1 --2- 172.21.15.29:0/639995951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56581017c0 0x7f565819e080 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5650008f10 tx=0x7f5650008f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:50.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.055+0000 7f563ffff700 1 -- 172.21.15.29:0/639995951 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f565000e040 con 0x7f56581017c0 2024-01-24T13:35:50.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.055+0000 7f565d1c7700 1 -- 172.21.15.29:0/639995951 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f565819eba0 con 0x7f56581017c0 2024-01-24T13:35:50.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.055+0000 7f563ffff700 1 -- 172.21.15.29:0/639995951 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5650005070 con 0x7f56581017c0 2024-01-24T13:35:50.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.055+0000 7f563ffff700 1 -- 172.21.15.29:0/639995951 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f565001b6a0 con 0x7f56581017c0 2024-01-24T13:35:50.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.055+0000 7f565d1c7700 1 -- 172.21.15.29:0/639995951 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f565819f020 con 0x7f56581017c0 2024-01-24T13:35:50.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.056+0000 7f565d1c7700 1 -- 172.21.15.29:0/639995951 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5658106ac0 con 0x7f56581017c0 2024-01-24T13:35:50.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.057+0000 7f563ffff700 1 -- 172.21.15.29:0/639995951 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f56500051e0 con 0x7f56581017c0 2024-01-24T13:35:50.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.058+0000 7f563ffff700 1 --2- 172.21.15.29:0/639995951 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f56400a0520 0x7f56400a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T13:35:50.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.058+0000 7f5656d9d700 1 --2- 172.21.15.29:0/639995951 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f56400a0520 0x7f56400a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T13:35:50.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.058+0000 7f563ffff700 1 -- 172.21.15.29:0/639995951 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f56500cd000 con 0x7f56581017c0 2024-01-24T13:35:50.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.058+0000 7f5656d9d700 1 --2- 172.21.15.29:0/639995951 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f56400a0520 0x7f56400a29e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5658100c80 tx=0x7f5648009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T13:35:50.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.061+0000 7f563ffff700 1 -- 172.21.15.29:0/639995951 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f5650096a40 con 0x7f56581017c0 2024-01-24T13:35:50.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:49 smithi163 ceph-mon[44969]: pgmap v234: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-01-24T13:35:50.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:49 smithi163 ceph-mon[44969]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-01-24T13:35:50.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:49 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/137184183' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-01-24T13:35:50.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:49 smithi163 ceph-mon[44969]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-01-24T13:35:50.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:49 smithi163 ceph-mon[44969]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-01-24T13:35:50.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:49 smithi163 ceph-mon[44969]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-01-24T13:35:50.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:49 smithi163 ceph-mon[44969]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-01-24T13:35:50.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:49 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/2083360077' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-01-24T13:35:50.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:49 smithi187 ceph-mon[44911]: pgmap v234: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-01-24T13:35:50.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:49 smithi187 ceph-mon[44911]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-01-24T13:35:50.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:49 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/137184183' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-01-24T13:35:50.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:49 smithi187 ceph-mon[44911]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-01-24T13:35:50.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:49 smithi187 ceph-mon[44911]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-01-24T13:35:50.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:49 smithi187 ceph-mon[44911]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-01-24T13:35:50.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:49 smithi187 ceph-mon[44911]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-01-24T13:35:50.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:49 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/2083360077' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-01-24T13:35:50.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.321+0000 7f565d1c7700 1 -- 172.21.15.29:0/639995951 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f565802cc70 con 0x7f56581017c0 2024-01-24T13:35:50.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:49 smithi029 ceph-mon[40651]: pgmap v234: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-01-24T13:35:50.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:49 smithi029 ceph-mon[40651]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-01-24T13:35:50.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:49 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/137184183' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-01-24T13:35:50.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:49 smithi029 ceph-mon[40651]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-01-24T13:35:50.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:49 smithi029 ceph-mon[40651]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-01-24T13:35:50.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:49 smithi029 ceph-mon[40651]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-01-24T13:35:50.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:49 smithi029 ceph-mon[40651]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-01-24T13:35:50.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:49 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/2083360077' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-01-24T13:35:50.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.839+0000 7f563ffff700 1 -- 172.21.15.29:0/639995951 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v22) v1 ==== 115+0+0 (secure 0 0 0) 0x7f565009a060 con 0x7f56581017c0 2024-01-24T13:35:50.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.842+0000 7f565d1c7700 1 -- 172.21.15.29:0/639995951 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f56400a0520 msgr2=0x7f56400a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:50.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.842+0000 7f565d1c7700 1 --2- 172.21.15.29:0/639995951 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f56400a0520 0x7f56400a29e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5658100c80 tx=0x7f5648009260 comp rx=0 tx=0).stop 2024-01-24T13:35:50.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.842+0000 7f565d1c7700 1 -- 172.21.15.29:0/639995951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56581017c0 msgr2=0x7f565819e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T13:35:50.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.842+0000 7f565d1c7700 1 --2- 172.21.15.29:0/639995951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56581017c0 0x7f565819e080 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5650008f10 tx=0x7f5650008f40 comp rx=0 tx=0).stop 2024-01-24T13:35:50.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.842+0000 7f565d1c7700 1 -- 172.21.15.29:0/639995951 shutdown_connections 2024-01-24T13:35:50.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.842+0000 7f565d1c7700 1 --2- 172.21.15.29:0/639995951 >> [v2:172.21.15.29:6800/2498991338,v1:172.21.15.29:6801/2498991338] conn(0x7f56400a0520 0x7f56400a29e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:50.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.842+0000 7f565d1c7700 1 --2- 172.21.15.29:0/639995951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56581017c0 0x7f565819e080 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:50.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.842+0000 7f565d1c7700 1 --2- 172.21.15.29:0/639995951 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f56580ffc30 0x7f5658196ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:50.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.842+0000 7f565d1c7700 1 --2- 172.21.15.29:0/639995951 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5658100e20 0x7f5658197010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T13:35:50.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.843+0000 7f565d1c7700 1 -- 172.21.15.29:0/639995951 >> 172.21.15.29:0/639995951 conn(0x7f56580fb1a0 msgr2=0x7f56580fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T13:35:50.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.843+0000 7f565d1c7700 1 -- 172.21.15.29:0/639995951 shutdown_connections 2024-01-24T13:35:50.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-24T13:35:50.843+0000 7f565d1c7700 1 -- 172.21.15.29:0/639995951 wait complete. 2024-01-24T13:35:50.860 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2024-01-24T13:35:50.860 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_4e3182a65800aed6d4aa2f6b86da3cfd72188134/qa/tasks/cephadm.py", line 1051, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/git.ceph.com_ceph-c_4e3182a65800aed6d4aa2f6b86da3cfd72188134/qa/tasks/cephfs/filesystem.py", line 532, in __init__ self.create() File "/home/teuthworker/src/git.ceph.com_ceph-c_4e3182a65800aed6d4aa2f6b86da3cfd72188134/qa/tasks/cephfs/filesystem.py", line 762, in create self.run_ceph_cmd('fs', 'subvolumegroup', 'create', self.name, 'qa') AttributeError: 'Filesystem' object has no attribute 'run_ceph_cmd' 2024-01-24T13:35:50.862 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-24T13:35:50.895 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-24T13:35:50.934 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-24T13:35:50.972 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-01-24T13:35:50.973 DEBUG:teuthology.orchestra.run.smithi029:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-24T13:35:50.994 DEBUG:teuthology.orchestra.run.smithi163:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-24T13:35:51.014 DEBUG:teuthology.orchestra.run.smithi187:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-24T13:35:51.033 INFO:tasks.cephadm:Stopping all daemons... 2024-01-24T13:35:51.033 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-01-24T13:35:51.033 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mon.a 2024-01-24T13:35:51.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:50 smithi163 ceph-mon[44969]: mds.? [v2:172.21.15.187:6838/3209015343,v1:172.21.15.187:6839/3209015343] up:active 2024-01-24T13:35:51.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:50 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/2083360077' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-01-24T13:35:51.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:50 smithi163 ceph-mon[44969]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-01-24T13:35:51.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:50 smithi163 ceph-mon[44969]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-01-24T13:35:51.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:50 smithi163 ceph-mon[44969]: Cluster is now healthy 2024-01-24T13:35:51.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:50 smithi163 ceph-mon[44969]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-01-24T13:35:51.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:50 smithi163 ceph-mon[44969]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-01-24T13:35:51.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:50 smithi163 ceph-mon[44969]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-01-24T13:35:51.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:50 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/639995951' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-01-24T13:35:51.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:50 smithi163 ceph-mon[44969]: mds.? [v2:172.21.15.163:6836/1610317406,v1:172.21.15.163:6837/1610317406] up:active 2024-01-24T13:35:51.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:50 smithi163 ceph-mon[44969]: from='client.? 172.21.15.29:0/639995951' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-01-24T13:35:51.233 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:50 smithi163 ceph-mon[44969]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-01-24T13:35:51.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:50 smithi187 ceph-mon[44911]: mds.? [v2:172.21.15.187:6838/3209015343,v1:172.21.15.187:6839/3209015343] up:active 2024-01-24T13:35:51.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:50 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/2083360077' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-01-24T13:35:51.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:50 smithi187 ceph-mon[44911]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-01-24T13:35:51.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:50 smithi187 ceph-mon[44911]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-01-24T13:35:51.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:50 smithi187 ceph-mon[44911]: Cluster is now healthy 2024-01-24T13:35:51.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:50 smithi187 ceph-mon[44911]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-01-24T13:35:51.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:50 smithi187 ceph-mon[44911]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-01-24T13:35:51.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:50 smithi187 ceph-mon[44911]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-01-24T13:35:51.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:50 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/639995951' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-01-24T13:35:51.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:50 smithi187 ceph-mon[44911]: mds.? [v2:172.21.15.163:6836/1610317406,v1:172.21.15.163:6837/1610317406] up:active 2024-01-24T13:35:51.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:50 smithi187 ceph-mon[44911]: from='client.? 172.21.15.29:0/639995951' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-01-24T13:35:51.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:50 smithi187 ceph-mon[44911]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-01-24T13:35:51.305 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:50 smithi029 ceph-mon[40651]: mds.? [v2:172.21.15.187:6838/3209015343,v1:172.21.15.187:6839/3209015343] up:active 2024-01-24T13:35:51.306 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:50 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/2083360077' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-01-24T13:35:51.306 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:50 smithi029 ceph-mon[40651]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-01-24T13:35:51.306 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:50 smithi029 ceph-mon[40651]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-01-24T13:35:51.306 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:50 smithi029 ceph-mon[40651]: Cluster is now healthy 2024-01-24T13:35:51.306 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:50 smithi029 ceph-mon[40651]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-01-24T13:35:51.306 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:50 smithi029 ceph-mon[40651]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-01-24T13:35:51.306 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:50 smithi029 ceph-mon[40651]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-01-24T13:35:51.306 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:50 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/639995951' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-01-24T13:35:51.306 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:50 smithi029 ceph-mon[40651]: mds.? [v2:172.21.15.163:6836/1610317406,v1:172.21.15.163:6837/1610317406] up:active 2024-01-24T13:35:51.306 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:50 smithi029 ceph-mon[40651]: from='client.? 172.21.15.29:0/639995951' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-01-24T13:35:51.306 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:50 smithi029 ceph-mon[40651]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-01-24T13:35:51.306 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:51 smithi029 systemd[1]: Stopping Ceph mon.a for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:35:51.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:51 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mon-a[40628]: 2024-01-24T13:35:51.302+0000 7f1f16663700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T13:35:51.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:51 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mon-a[40628]: 2024-01-24T13:35:51.302+0000 7f1f16663700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-01-24T13:35:51.968 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jan 24 13:35:51 smithi029 bash[96079]: ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mon-a 2024-01-24T13:35:51.975 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mon.a.service' 2024-01-24T13:35:52.031 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T13:35:52.031 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-01-24T13:35:52.032 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-01-24T13:35:52.032 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mon.b 2024-01-24T13:35:52.352 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:52 smithi163 systemd[1]: Stopping Ceph mon.b for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:35:52.352 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:52 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mon-b[44946]: 2024-01-24T13:35:52.350+0000 7fc5db2ee700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T13:35:52.352 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:52 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mon-b[44946]: 2024-01-24T13:35:52.350+0000 7fc5db2ee700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-01-24T13:35:52.732 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jan 24 13:35:52 smithi163 bash[84229]: ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mon-b 2024-01-24T13:35:52.866 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mon.b.service' 2024-01-24T13:35:52.923 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T13:35:52.923 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-01-24T13:35:52.923 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-01-24T13:35:52.923 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mon.c 2024-01-24T13:35:53.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:52 smithi187 systemd[1]: Stopping Ceph mon.c for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:35:53.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:53 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mon-c[44888]: 2024-01-24T13:35:53.144+0000 7f483b8eb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T13:35:53.305 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:53 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mon-c[44888]: 2024-01-24T13:35:53.144+0000 7f483b8eb700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-01-24T13:35:53.804 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jan 24 13:35:53 smithi187 bash[83108]: ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mon-c 2024-01-24T13:35:53.848 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mon.c.service' 2024-01-24T13:35:53.900 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T13:35:53.900 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-01-24T13:35:53.900 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-01-24T13:35:53.900 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mgr.x 2024-01-24T13:35:54.189 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:35:53 smithi029 systemd[1]: Stopping Ceph mgr.x for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:35:54.599 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jan 24 13:35:54 smithi029 bash[96574]: ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-x 2024-01-24T13:35:54.713 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mgr.x.service' 2024-01-24T13:35:54.758 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T13:35:54.758 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-01-24T13:35:54.758 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-01-24T13:35:54.758 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mgr.y 2024-01-24T13:35:55.137 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Jan 24 13:35:54 smithi163 systemd[1]: Stopping Ceph mgr.y for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:35:55.482 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Jan 24 13:35:55 smithi163 bash[84433]: ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-y 2024-01-24T13:35:55.482 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Jan 24 13:35:55 smithi163 systemd[1]: ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-01-24T13:35:55.514 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mgr.y.service' 2024-01-24T13:35:55.551 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T13:35:55.552 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-01-24T13:35:55.552 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-01-24T13:35:55.552 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mgr.z 2024-01-24T13:35:55.925 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jan 24 13:35:55 smithi187 systemd[1]: Stopping Ceph mgr.z for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:35:56.260 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jan 24 13:35:55 smithi187 bash[83257]: ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mgr-z 2024-01-24T13:35:56.260 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jan 24 13:35:56 smithi187 systemd[1]: ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-01-24T13:35:56.267 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mgr.z.service' 2024-01-24T13:35:56.303 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T13:35:56.303 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-01-24T13:35:56.304 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-01-24T13:35:56.304 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-83afcf5e-babc-11ee-95b1-87774f69a715@osd.0 2024-01-24T13:35:56.599 INFO:journalctl@ceph.osd.0.smithi029.stdout:Jan 24 13:35:56 smithi029 systemd[1]: Stopping Ceph osd.0 for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:35:56.599 INFO:journalctl@ceph.osd.0.smithi029.stdout:Jan 24 13:35:56 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-0[59238]: 2024-01-24T13:35:56.568+0000 7f60656bc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T13:35:56.599 INFO:journalctl@ceph.osd.0.smithi029.stdout:Jan 24 13:35:56 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-0[59238]: 2024-01-24T13:35:56.568+0000 7f60656bc700 -1 osd.0 85 *** Got signal Terminated *** 2024-01-24T13:35:56.599 INFO:journalctl@ceph.osd.0.smithi029.stdout:Jan 24 13:35:56 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-0[59238]: 2024-01-24T13:35:56.568+0000 7f60656bc700 -1 osd.0 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-24T13:36:02.349 INFO:journalctl@ceph.osd.0.smithi029.stdout:Jan 24 13:36:02 smithi029 bash[96714]: ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-0 2024-01-24T13:36:03.451 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@osd.0.service' 2024-01-24T13:36:03.501 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T13:36:03.502 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-01-24T13:36:03.502 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-01-24T13:36:03.502 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-83afcf5e-babc-11ee-95b1-87774f69a715@osd.1 2024-01-24T13:36:03.982 INFO:journalctl@ceph.osd.1.smithi163.stdout:Jan 24 13:36:03 smithi163 systemd[1]: Stopping Ceph osd.1 for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:36:03.982 INFO:journalctl@ceph.osd.1.smithi163.stdout:Jan 24 13:36:03 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-1[49943]: 2024-01-24T13:36:03.748+0000 7f77bfa26700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T13:36:03.982 INFO:journalctl@ceph.osd.1.smithi163.stdout:Jan 24 13:36:03 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-1[49943]: 2024-01-24T13:36:03.748+0000 7f77bfa26700 -1 osd.1 85 *** Got signal Terminated *** 2024-01-24T13:36:03.982 INFO:journalctl@ceph.osd.1.smithi163.stdout:Jan 24 13:36:03 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-1[49943]: 2024-01-24T13:36:03.748+0000 7f77bfa26700 -1 osd.1 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-24T13:36:09.732 INFO:journalctl@ceph.osd.1.smithi163.stdout:Jan 24 13:36:09 smithi163 bash[84572]: ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-1 2024-01-24T13:36:10.594 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@osd.1.service' 2024-01-24T13:36:10.634 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T13:36:10.634 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-01-24T13:36:10.635 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-01-24T13:36:10.635 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-83afcf5e-babc-11ee-95b1-87774f69a715@osd.2 2024-01-24T13:36:11.055 INFO:journalctl@ceph.osd.2.smithi187.stdout:Jan 24 13:36:10 smithi187 systemd[1]: Stopping Ceph osd.2 for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:36:11.055 INFO:journalctl@ceph.osd.2.smithi187.stdout:Jan 24 13:36:10 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-2[50708]: 2024-01-24T13:36:10.883+0000 7f07cb50c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T13:36:11.055 INFO:journalctl@ceph.osd.2.smithi187.stdout:Jan 24 13:36:10 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-2[50708]: 2024-01-24T13:36:10.883+0000 7f07cb50c700 -1 osd.2 85 *** Got signal Terminated *** 2024-01-24T13:36:11.055 INFO:journalctl@ceph.osd.2.smithi187.stdout:Jan 24 13:36:10 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-2[50708]: 2024-01-24T13:36:10.883+0000 7f07cb50c700 -1 osd.2 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-24T13:36:16.805 INFO:journalctl@ceph.osd.2.smithi187.stdout:Jan 24 13:36:16 smithi187 bash[83411]: ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-2 2024-01-24T13:36:17.754 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@osd.2.service' 2024-01-24T13:36:17.796 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T13:36:17.797 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-01-24T13:36:17.797 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-01-24T13:36:17.797 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-83afcf5e-babc-11ee-95b1-87774f69a715@osd.3 2024-01-24T13:36:18.099 INFO:journalctl@ceph.osd.3.smithi029.stdout:Jan 24 13:36:17 smithi029 systemd[1]: Stopping Ceph osd.3 for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:36:18.099 INFO:journalctl@ceph.osd.3.smithi029.stdout:Jan 24 13:36:18 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-3[67880]: 2024-01-24T13:36:18.041+0000 7f4a6605b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T13:36:18.099 INFO:journalctl@ceph.osd.3.smithi029.stdout:Jan 24 13:36:18 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-3[67880]: 2024-01-24T13:36:18.041+0000 7f4a6605b700 -1 osd.3 85 *** Got signal Terminated *** 2024-01-24T13:36:18.099 INFO:journalctl@ceph.osd.3.smithi029.stdout:Jan 24 13:36:18 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-3[67880]: 2024-01-24T13:36:18.041+0000 7f4a6605b700 -1 osd.3 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-24T13:36:23.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:22 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:22.589+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.895912+0000 front 2024-01-24T13:35:57.895940+0000 (oldest deadline 2024-01-24T13:36:21.995938+0000) 2024-01-24T13:36:23.849 INFO:journalctl@ceph.osd.3.smithi029.stdout:Jan 24 13:36:23 smithi029 bash[96972]: ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-3 2024-01-24T13:36:23.888 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:23 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:23.637+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.895912+0000 front 2024-01-24T13:35:57.895940+0000 (oldest deadline 2024-01-24T13:36:21.995938+0000) 2024-01-24T13:36:24.187 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jan 24 13:36:23 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-5[58508]: 2024-01-24T13:36:23.888+0000 7fdaf57e1700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.543218+0000 front 2024-01-24T13:35:57.543364+0000 (oldest deadline 2024-01-24T13:36:23.443271+0000) 2024-01-24T13:36:24.522 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:24 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:24.235+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:36:24.554 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:24 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:24.186+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:36:24.856 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@osd.3.service' 2024-01-24T13:36:24.908 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T13:36:24.908 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-01-24T13:36:24.908 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-01-24T13:36:24.908 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-83afcf5e-babc-11ee-95b1-87774f69a715@osd.4 2024-01-24T13:36:25.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:24 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:24.630+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.895912+0000 front 2024-01-24T13:35:57.895940+0000 (oldest deadline 2024-01-24T13:36:21.995938+0000) 2024-01-24T13:36:25.054 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jan 24 13:36:24 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-5[58508]: 2024-01-24T13:36:24.851+0000 7fdaf57e1700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.543218+0000 front 2024-01-24T13:35:57.543364+0000 (oldest deadline 2024-01-24T13:36:23.443271+0000) 2024-01-24T13:36:25.188 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:24 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:24.943+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:58.893086+0000 front 2024-01-24T13:35:58.893135+0000 (oldest deadline 2024-01-24T13:36:24.693987+0000) 2024-01-24T13:36:25.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:24 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:24.994+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:36:25.232 INFO:journalctl@ceph.osd.4.smithi163.stdout:Jan 24 13:36:24 smithi163 systemd[1]: Stopping Ceph osd.4 for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:36:25.232 INFO:journalctl@ceph.osd.4.smithi163.stdout:Jan 24 13:36:25 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-4[58614]: 2024-01-24T13:36:25.151+0000 7f5b746a0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T13:36:25.232 INFO:journalctl@ceph.osd.4.smithi163.stdout:Jan 24 13:36:25 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-4[58614]: 2024-01-24T13:36:25.151+0000 7f5b746a0700 -1 osd.4 85 *** Got signal Terminated *** 2024-01-24T13:36:25.232 INFO:journalctl@ceph.osd.4.smithi163.stdout:Jan 24 13:36:25 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-4[58614]: 2024-01-24T13:36:25.151+0000 7f5b746a0700 -1 osd.4 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-24T13:36:25.554 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:25 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:25.185+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:36:25.598 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:25 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:25.188+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:36:26.054 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jan 24 13:36:25 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-5[58508]: 2024-01-24T13:36:25.850+0000 7fdaf57e1700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.543218+0000 front 2024-01-24T13:35:57.543364+0000 (oldest deadline 2024-01-24T13:36:23.443271+0000) 2024-01-24T13:36:26.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:25 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:25.610+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.895912+0000 front 2024-01-24T13:35:57.895940+0000 (oldest deadline 2024-01-24T13:36:21.995938+0000) 2024-01-24T13:36:26.211 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:25 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:25.906+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:58.893086+0000 front 2024-01-24T13:35:58.893135+0000 (oldest deadline 2024-01-24T13:36:24.693987+0000) 2024-01-24T13:36:26.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:26 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:26.029+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:36:26.554 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:26 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:26.171+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:36:26.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:26 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:26.211+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:36:27.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:26 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:26.658+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.895912+0000 front 2024-01-24T13:35:57.895940+0000 (oldest deadline 2024-01-24T13:36:21.995938+0000) 2024-01-24T13:36:27.055 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jan 24 13:36:26 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-5[58508]: 2024-01-24T13:36:26.847+0000 7fdaf57e1700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.543218+0000 front 2024-01-24T13:35:57.543364+0000 (oldest deadline 2024-01-24T13:36:23.443271+0000) 2024-01-24T13:36:27.171 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:26 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:26.887+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:58.893086+0000 front 2024-01-24T13:35:58.893135+0000 (oldest deadline 2024-01-24T13:36:24.693987+0000) 2024-01-24T13:36:27.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:26 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:26.988+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:36:27.554 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:27 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:27.139+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:36:27.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:27 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:27.171+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:36:28.054 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jan 24 13:36:27 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-5[58508]: 2024-01-24T13:36:27.806+0000 7fdaf57e1700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.543218+0000 front 2024-01-24T13:35:57.543364+0000 (oldest deadline 2024-01-24T13:36:23.443271+0000) 2024-01-24T13:36:28.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:27 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:27.701+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.895912+0000 front 2024-01-24T13:35:57.895940+0000 (oldest deadline 2024-01-24T13:36:21.995938+0000) 2024-01-24T13:36:28.205 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:27 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:27.863+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:58.893086+0000 front 2024-01-24T13:35:58.893135+0000 (oldest deadline 2024-01-24T13:36:24.693987+0000) 2024-01-24T13:36:28.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:27 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:27.962+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:36:28.554 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:28 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:28.160+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:36:28.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:28 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:28.204+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:36:29.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:28 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:28.679+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.895912+0000 front 2024-01-24T13:35:57.895940+0000 (oldest deadline 2024-01-24T13:36:21.995938+0000) 2024-01-24T13:36:29.055 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jan 24 13:36:28 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-5[58508]: 2024-01-24T13:36:28.781+0000 7fdaf57e1700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.543218+0000 front 2024-01-24T13:35:57.543364+0000 (oldest deadline 2024-01-24T13:36:23.443271+0000) 2024-01-24T13:36:29.099 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:28 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:28.827+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:58.893086+0000 front 2024-01-24T13:35:58.893135+0000 (oldest deadline 2024-01-24T13:36:24.693987+0000) 2024-01-24T13:36:29.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:28 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:28.967+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:36:29.554 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:29 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:29.203+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:36:29.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:29 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:29.211+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:36:30.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:29 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:29.653+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.895912+0000 front 2024-01-24T13:35:57.895940+0000 (oldest deadline 2024-01-24T13:36:21.995938+0000) 2024-01-24T13:36:30.055 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jan 24 13:36:29 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-5[58508]: 2024-01-24T13:36:29.795+0000 7fdaf57e1700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.543218+0000 front 2024-01-24T13:35:57.543364+0000 (oldest deadline 2024-01-24T13:36:23.443271+0000) 2024-01-24T13:36:30.099 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:29 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:29.823+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:58.893086+0000 front 2024-01-24T13:35:58.893135+0000 (oldest deadline 2024-01-24T13:36:24.693987+0000) 2024-01-24T13:36:30.099 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:29 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:29.823+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:04.694619+0000 front 2024-01-24T13:36:04.694633+0000 (oldest deadline 2024-01-24T13:36:29.394613+0000) 2024-01-24T13:36:30.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:29 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:29.946+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:36:30.554 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:30 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:30.177+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:36:30.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:30 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:30.221+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:36:30.982 INFO:journalctl@ceph.osd.4.smithi163.stdout:Jan 24 13:36:30 smithi163 bash[84841]: ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-4 2024-01-24T13:36:30.983 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:30 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:30.669+0000 7f814fc80700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:05.126625+0000 front 2024-01-24T13:36:05.126588+0000 (oldest deadline 2024-01-24T13:36:29.826734+0000) 2024-01-24T13:36:30.983 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:30 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:30.906+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:36:31.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:30 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:30.679+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.895912+0000 front 2024-01-24T13:35:57.895940+0000 (oldest deadline 2024-01-24T13:36:21.995938+0000) 2024-01-24T13:36:31.054 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jan 24 13:36:30 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-5[58508]: 2024-01-24T13:36:30.842+0000 7fdaf57e1700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.543218+0000 front 2024-01-24T13:35:57.543364+0000 (oldest deadline 2024-01-24T13:36:23.443271+0000) 2024-01-24T13:36:31.179 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:30 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:30.860+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:58.893086+0000 front 2024-01-24T13:35:58.893135+0000 (oldest deadline 2024-01-24T13:36:24.693987+0000) 2024-01-24T13:36:31.179 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:30 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:30.860+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:04.694619+0000 front 2024-01-24T13:36:04.694633+0000 (oldest deadline 2024-01-24T13:36:29.394613+0000) 2024-01-24T13:36:31.554 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:31 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:31.192+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:36:31.598 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:31 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:31.178+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:36:31.943 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:31 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:31.687+0000 7f814fc80700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:05.126625+0000 front 2024-01-24T13:36:05.126588+0000 (oldest deadline 2024-01-24T13:36:29.826734+0000) 2024-01-24T13:36:31.972 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@osd.4.service' 2024-01-24T13:36:32.013 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T13:36:32.013 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-01-24T13:36:32.013 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-01-24T13:36:32.014 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-83afcf5e-babc-11ee-95b1-87774f69a715@osd.5 2024-01-24T13:36:32.044 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:31 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:31.695+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.895912+0000 front 2024-01-24T13:35:57.895940+0000 (oldest deadline 2024-01-24T13:36:21.995938+0000) 2024-01-24T13:36:32.044 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jan 24 13:36:31 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-5[58508]: 2024-01-24T13:36:31.794+0000 7fdaf57e1700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.543218+0000 front 2024-01-24T13:35:57.543364+0000 (oldest deadline 2024-01-24T13:36:23.443271+0000) 2024-01-24T13:36:32.129 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:31 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:31.861+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:58.893086+0000 front 2024-01-24T13:35:58.893135+0000 (oldest deadline 2024-01-24T13:36:24.693987+0000) 2024-01-24T13:36:32.129 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:31 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:31.861+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:04.694619+0000 front 2024-01-24T13:36:04.694633+0000 (oldest deadline 2024-01-24T13:36:29.394613+0000) 2024-01-24T13:36:32.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:31 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:31.942+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:36:32.305 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:32 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:32.163+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:36:32.305 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jan 24 13:36:32 smithi187 systemd[1]: Stopping Ceph osd.5 for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:36:32.305 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jan 24 13:36:32 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-5[58508]: 2024-01-24T13:36:32.255+0000 7fdafd1f3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T13:36:32.305 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jan 24 13:36:32 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-5[58508]: 2024-01-24T13:36:32.255+0000 7fdafd1f3700 -1 osd.5 85 *** Got signal Terminated *** 2024-01-24T13:36:32.305 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jan 24 13:36:32 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-5[58508]: 2024-01-24T13:36:32.255+0000 7fdafd1f3700 -1 osd.5 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-24T13:36:32.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:32 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:32.128+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:36:32.963 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:32 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:32.656+0000 7f814fc80700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:05.126625+0000 front 2024-01-24T13:36:05.126588+0000 (oldest deadline 2024-01-24T13:36:29.826734+0000) 2024-01-24T13:36:33.054 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jan 24 13:36:32 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-5[58508]: 2024-01-24T13:36:32.746+0000 7fdaf57e1700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.543218+0000 front 2024-01-24T13:35:57.543364+0000 (oldest deadline 2024-01-24T13:36:23.443271+0000) 2024-01-24T13:36:33.054 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jan 24 13:36:32 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-5[58508]: 2024-01-24T13:36:32.746+0000 7fdaf57e1700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:06.944557+0000 front 2024-01-24T13:36:06.944387+0000 (oldest deadline 2024-01-24T13:36:32.244340+0000) 2024-01-24T13:36:33.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:32 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:32.704+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.895912+0000 front 2024-01-24T13:35:57.895940+0000 (oldest deadline 2024-01-24T13:36:21.995938+0000) 2024-01-24T13:36:33.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:32 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:32.704+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.197664+0000 front 2024-01-24T13:36:07.197789+0000 (oldest deadline 2024-01-24T13:36:31.897749+0000) 2024-01-24T13:36:33.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:32 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:32.961+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:36:33.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:32 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:32.961+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:36:33.349 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:32 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:32.890+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:58.893086+0000 front 2024-01-24T13:35:58.893135+0000 (oldest deadline 2024-01-24T13:36:24.693987+0000) 2024-01-24T13:36:33.349 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:32 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:32.890+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:04.694619+0000 front 2024-01-24T13:36:04.694633+0000 (oldest deadline 2024-01-24T13:36:29.394613+0000) 2024-01-24T13:36:33.349 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:33 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:33.112+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:36:33.349 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:33 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:33.112+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.684218+0000 front 2024-01-24T13:36:08.684295+0000 (oldest deadline 2024-01-24T13:36:32.784244+0000) 2024-01-24T13:36:33.554 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:33 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:33.198+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:36:33.554 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:33 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:33.198+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:36:33.931 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:33 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:33.607+0000 7f814fc80700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:05.126625+0000 front 2024-01-24T13:36:05.126588+0000 (oldest deadline 2024-01-24T13:36:29.826734+0000) 2024-01-24T13:36:34.054 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jan 24 13:36:33 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-5[58508]: 2024-01-24T13:36:33.752+0000 7fdaf57e1700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.543218+0000 front 2024-01-24T13:35:57.543364+0000 (oldest deadline 2024-01-24T13:36:23.443271+0000) 2024-01-24T13:36:34.055 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jan 24 13:36:33 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-5[58508]: 2024-01-24T13:36:33.752+0000 7fdaf57e1700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:06.944557+0000 front 2024-01-24T13:36:06.944387+0000 (oldest deadline 2024-01-24T13:36:32.244340+0000) 2024-01-24T13:36:34.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:33 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:33.732+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.895912+0000 front 2024-01-24T13:35:57.895940+0000 (oldest deadline 2024-01-24T13:36:21.995938+0000) 2024-01-24T13:36:34.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:33 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:33.732+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.197664+0000 front 2024-01-24T13:36:07.197789+0000 (oldest deadline 2024-01-24T13:36:31.897749+0000) 2024-01-24T13:36:34.126 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:33 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:33.862+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:58.893086+0000 front 2024-01-24T13:35:58.893135+0000 (oldest deadline 2024-01-24T13:36:24.693987+0000) 2024-01-24T13:36:34.126 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:33 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:33.862+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:04.694619+0000 front 2024-01-24T13:36:04.694633+0000 (oldest deadline 2024-01-24T13:36:29.394613+0000) 2024-01-24T13:36:34.231 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:33 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:33.931+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:36:34.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:33 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:33.931+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:36:34.554 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:34 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:34.192+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:36:34.554 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:34 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:34.192+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:36:34.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:34 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:34.125+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:36:34.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:34 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:34.125+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.684218+0000 front 2024-01-24T13:36:08.684295+0000 (oldest deadline 2024-01-24T13:36:32.784244+0000) 2024-01-24T13:36:34.936 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:34 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:34.623+0000 7f814fc80700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:05.126625+0000 front 2024-01-24T13:36:05.126588+0000 (oldest deadline 2024-01-24T13:36:29.826734+0000) 2024-01-24T13:36:35.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:34 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:34.698+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.895912+0000 front 2024-01-24T13:35:57.895940+0000 (oldest deadline 2024-01-24T13:36:21.995938+0000) 2024-01-24T13:36:35.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:34 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:34.698+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.197664+0000 front 2024-01-24T13:36:07.197789+0000 (oldest deadline 2024-01-24T13:36:31.897749+0000) 2024-01-24T13:36:35.055 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jan 24 13:36:34 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-5[58508]: 2024-01-24T13:36:34.766+0000 7fdaf57e1700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.543218+0000 front 2024-01-24T13:35:57.543364+0000 (oldest deadline 2024-01-24T13:36:23.443271+0000) 2024-01-24T13:36:35.055 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jan 24 13:36:34 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-5[58508]: 2024-01-24T13:36:34.766+0000 7fdaf57e1700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:06.944557+0000 front 2024-01-24T13:36:06.944387+0000 (oldest deadline 2024-01-24T13:36:32.244340+0000) 2024-01-24T13:36:35.147 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:34 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:34.889+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:58.893086+0000 front 2024-01-24T13:35:58.893135+0000 (oldest deadline 2024-01-24T13:36:24.693987+0000) 2024-01-24T13:36:35.147 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:34 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:34.889+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:04.694619+0000 front 2024-01-24T13:36:04.694633+0000 (oldest deadline 2024-01-24T13:36:29.394613+0000) 2024-01-24T13:36:35.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:34 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:34.936+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:36:35.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:34 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:34.936+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:36:35.554 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:35 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:35.234+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:36:35.555 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:35 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:35.234+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:36:35.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:35 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:35.146+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:36:35.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:35 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:35.146+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.684218+0000 front 2024-01-24T13:36:08.684295+0000 (oldest deadline 2024-01-24T13:36:32.784244+0000) 2024-01-24T13:36:35.953 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:35 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:35.667+0000 7f814fc80700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:05.126625+0000 front 2024-01-24T13:36:05.126588+0000 (oldest deadline 2024-01-24T13:36:29.826734+0000) 2024-01-24T13:36:36.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:35 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:35.658+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.895912+0000 front 2024-01-24T13:35:57.895940+0000 (oldest deadline 2024-01-24T13:36:21.995938+0000) 2024-01-24T13:36:36.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:35 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:35.659+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.197664+0000 front 2024-01-24T13:36:07.197789+0000 (oldest deadline 2024-01-24T13:36:31.897749+0000) 2024-01-24T13:36:36.055 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jan 24 13:36:35 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-5[58508]: 2024-01-24T13:36:35.795+0000 7fdaf57e1700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.543218+0000 front 2024-01-24T13:35:57.543364+0000 (oldest deadline 2024-01-24T13:36:23.443271+0000) 2024-01-24T13:36:36.055 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jan 24 13:36:35 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-5[58508]: 2024-01-24T13:36:35.795+0000 7fdaf57e1700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:06.944557+0000 front 2024-01-24T13:36:06.944387+0000 (oldest deadline 2024-01-24T13:36:32.244340+0000) 2024-01-24T13:36:36.156 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:35 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:35.882+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:58.893086+0000 front 2024-01-24T13:35:58.893135+0000 (oldest deadline 2024-01-24T13:36:24.693987+0000) 2024-01-24T13:36:36.156 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:35 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:35.882+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:04.694619+0000 front 2024-01-24T13:36:04.694633+0000 (oldest deadline 2024-01-24T13:36:29.394613+0000) 2024-01-24T13:36:36.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:35 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:35.953+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:36:36.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:35 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:35.953+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:36:36.554 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:36 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:36.280+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:36:36.554 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:36 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:36.280+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:36:36.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:36 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:36.155+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:36:36.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:36 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:36.155+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.684218+0000 front 2024-01-24T13:36:08.684295+0000 (oldest deadline 2024-01-24T13:36:32.784244+0000) 2024-01-24T13:36:36.982 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:36 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:36.712+0000 7f814fc80700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:05.126625+0000 front 2024-01-24T13:36:05.126588+0000 (oldest deadline 2024-01-24T13:36:29.826734+0000) 2024-01-24T13:36:36.982 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:36 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:36.956+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:36:36.982 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:36 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:36.956+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:36:37.054 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jan 24 13:36:36 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-5[58508]: 2024-01-24T13:36:36.813+0000 7fdaf57e1700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.543218+0000 front 2024-01-24T13:35:57.543364+0000 (oldest deadline 2024-01-24T13:36:23.443271+0000) 2024-01-24T13:36:37.055 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jan 24 13:36:36 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-5[58508]: 2024-01-24T13:36:36.813+0000 7fdaf57e1700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:06.944557+0000 front 2024-01-24T13:36:06.944387+0000 (oldest deadline 2024-01-24T13:36:32.244340+0000) 2024-01-24T13:36:37.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:36 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:36.637+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.895912+0000 front 2024-01-24T13:35:57.895940+0000 (oldest deadline 2024-01-24T13:36:21.995938+0000) 2024-01-24T13:36:37.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:36 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:36.637+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.197664+0000 front 2024-01-24T13:36:07.197789+0000 (oldest deadline 2024-01-24T13:36:31.897749+0000) 2024-01-24T13:36:37.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:36 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:36.637+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:13.598791+0000 front 2024-01-24T13:36:13.598854+0000 (oldest deadline 2024-01-24T13:36:36.499036+0000) 2024-01-24T13:36:37.349 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:36 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:36.875+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:58.893086+0000 front 2024-01-24T13:35:58.893135+0000 (oldest deadline 2024-01-24T13:36:24.693987+0000) 2024-01-24T13:36:37.349 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:36 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:36.875+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:04.694619+0000 front 2024-01-24T13:36:04.694633+0000 (oldest deadline 2024-01-24T13:36:29.394613+0000) 2024-01-24T13:36:37.349 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:37 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:37.119+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:36:37.349 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:37 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:37.119+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.684218+0000 front 2024-01-24T13:36:08.684295+0000 (oldest deadline 2024-01-24T13:36:32.784244+0000) 2024-01-24T13:36:37.647 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:37 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:37.322+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:36:37.647 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:37 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:37.322+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:36:37.647 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:37 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:37.322+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:36:37.982 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:37 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:37.712+0000 7f814fc80700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:05.126625+0000 front 2024-01-24T13:36:05.126588+0000 (oldest deadline 2024-01-24T13:36:29.826734+0000) 2024-01-24T13:36:38.054 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jan 24 13:36:37 smithi187 bash[83667]: ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-5 2024-01-24T13:36:38.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:37 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:37.647+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.895912+0000 front 2024-01-24T13:35:57.895940+0000 (oldest deadline 2024-01-24T13:36:21.995938+0000) 2024-01-24T13:36:38.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:37 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:37.647+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.197664+0000 front 2024-01-24T13:36:07.197789+0000 (oldest deadline 2024-01-24T13:36:31.897749+0000) 2024-01-24T13:36:38.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:37 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:37.647+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:13.598791+0000 front 2024-01-24T13:36:13.598854+0000 (oldest deadline 2024-01-24T13:36:36.499036+0000) 2024-01-24T13:36:38.349 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:37 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:37.873+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:58.893086+0000 front 2024-01-24T13:35:58.893135+0000 (oldest deadline 2024-01-24T13:36:24.693987+0000) 2024-01-24T13:36:38.349 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:37 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:37.873+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:04.694619+0000 front 2024-01-24T13:36:04.694633+0000 (oldest deadline 2024-01-24T13:36:29.394613+0000) 2024-01-24T13:36:38.349 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:37 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:37.873+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:13.396327+0000 front 2024-01-24T13:36:13.396184+0000 (oldest deadline 2024-01-24T13:36:36.896245+0000) 2024-01-24T13:36:38.349 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:38 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:38.108+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:36:38.349 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:38 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:38.108+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.684218+0000 front 2024-01-24T13:36:08.684295+0000 (oldest deadline 2024-01-24T13:36:32.784244+0000) 2024-01-24T13:36:38.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:37 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:37.992+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:36:38.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:37 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:37.992+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:36:38.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:37 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:37.992+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.121624+0000 front 2024-01-24T13:36:12.121460+0000 (oldest deadline 2024-01-24T13:36:37.321997+0000) 2024-01-24T13:36:38.554 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:38 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:38.292+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:36:38.554 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:38 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:38.292+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:36:38.554 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:38 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:38.292+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:36:39.042 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:38 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:38.663+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.895912+0000 front 2024-01-24T13:35:57.895940+0000 (oldest deadline 2024-01-24T13:36:21.995938+0000) 2024-01-24T13:36:39.043 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:38 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:38.663+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.197664+0000 front 2024-01-24T13:36:07.197789+0000 (oldest deadline 2024-01-24T13:36:31.897749+0000) 2024-01-24T13:36:39.043 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:38 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:38.663+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:13.598791+0000 front 2024-01-24T13:36:13.598854+0000 (oldest deadline 2024-01-24T13:36:36.499036+0000) 2024-01-24T13:36:39.050 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@osd.5.service' 2024-01-24T13:36:39.110 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:38 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:38.854+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:58.893086+0000 front 2024-01-24T13:35:58.893135+0000 (oldest deadline 2024-01-24T13:36:24.693987+0000) 2024-01-24T13:36:39.110 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:38 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:38.854+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:04.694619+0000 front 2024-01-24T13:36:04.694633+0000 (oldest deadline 2024-01-24T13:36:29.394613+0000) 2024-01-24T13:36:39.110 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:38 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:38.854+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:13.396327+0000 front 2024-01-24T13:36:13.396184+0000 (oldest deadline 2024-01-24T13:36:36.896245+0000) 2024-01-24T13:36:39.133 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T13:36:39.133 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-01-24T13:36:39.134 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-01-24T13:36:39.134 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-83afcf5e-babc-11ee-95b1-87774f69a715@osd.6 2024-01-24T13:36:39.232 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:38 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:38.758+0000 7f814fc80700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:05.126625+0000 front 2024-01-24T13:36:05.126588+0000 (oldest deadline 2024-01-24T13:36:29.826734+0000) 2024-01-24T13:36:39.232 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:38 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:38.758+0000 7f814fc80700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.628400+0000 front 2024-01-24T13:36:12.628490+0000 (oldest deadline 2024-01-24T13:36:38.528258+0000) 2024-01-24T13:36:39.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:38 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:38.980+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:36:39.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:38 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:38.980+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:36:39.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:38 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:38.980+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.121624+0000 front 2024-01-24T13:36:12.121460+0000 (oldest deadline 2024-01-24T13:36:37.321997+0000) 2024-01-24T13:36:39.389 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:39 smithi029 systemd[1]: Stopping Ceph osd.6 for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:36:39.389 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:39 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:39.110+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:36:39.390 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:39 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:39.110+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.684218+0000 front 2024-01-24T13:36:08.684295+0000 (oldest deadline 2024-01-24T13:36:32.784244+0000) 2024-01-24T13:36:39.390 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:39 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:39.110+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.784926+0000 front 2024-01-24T13:36:12.784800+0000 (oldest deadline 2024-01-24T13:36:38.684866+0000) 2024-01-24T13:36:39.666 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:39 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:39.310+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:36:39.667 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:39 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:39.310+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:36:39.667 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:39 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:39.310+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:36:39.826 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:39 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:39.388+0000 7f858f352700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T13:36:39.826 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:39 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:39.388+0000 7f858f352700 -1 osd.6 85 *** Got signal Terminated *** 2024-01-24T13:36:39.826 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:39 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:39.388+0000 7f858f352700 -1 osd.6 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-24T13:36:39.982 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:39 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:39.721+0000 7f814fc80700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:05.126625+0000 front 2024-01-24T13:36:05.126588+0000 (oldest deadline 2024-01-24T13:36:29.826734+0000) 2024-01-24T13:36:39.982 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:39 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:39.721+0000 7f814fc80700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.628400+0000 front 2024-01-24T13:36:12.628490+0000 (oldest deadline 2024-01-24T13:36:38.528258+0000) 2024-01-24T13:36:40.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:39 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:39.667+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.895912+0000 front 2024-01-24T13:35:57.895940+0000 (oldest deadline 2024-01-24T13:36:21.995938+0000) 2024-01-24T13:36:40.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:39 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:39.667+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.197664+0000 front 2024-01-24T13:36:07.197789+0000 (oldest deadline 2024-01-24T13:36:31.897749+0000) 2024-01-24T13:36:40.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:39 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:39.667+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:13.598791+0000 front 2024-01-24T13:36:13.598854+0000 (oldest deadline 2024-01-24T13:36:36.499036+0000) 2024-01-24T13:36:40.080 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:39 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:39.825+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:58.893086+0000 front 2024-01-24T13:35:58.893135+0000 (oldest deadline 2024-01-24T13:36:24.693987+0000) 2024-01-24T13:36:40.080 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:39 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:39.825+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:04.694619+0000 front 2024-01-24T13:36:04.694633+0000 (oldest deadline 2024-01-24T13:36:29.394613+0000) 2024-01-24T13:36:40.081 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:39 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:39.825+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:13.396327+0000 front 2024-01-24T13:36:13.396184+0000 (oldest deadline 2024-01-24T13:36:36.896245+0000) 2024-01-24T13:36:40.349 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:40 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:40.080+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:36:40.349 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:40 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:40.080+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.684218+0000 front 2024-01-24T13:36:08.684295+0000 (oldest deadline 2024-01-24T13:36:32.784244+0000) 2024-01-24T13:36:40.349 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:40 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:40.080+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.784926+0000 front 2024-01-24T13:36:12.784800+0000 (oldest deadline 2024-01-24T13:36:38.684866+0000) 2024-01-24T13:36:40.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:39 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:39.993+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:36:40.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:39 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:39.993+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:36:40.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:39 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:39.993+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.121624+0000 front 2024-01-24T13:36:12.121460+0000 (oldest deadline 2024-01-24T13:36:37.321997+0000) 2024-01-24T13:36:40.661 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:40 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:40.326+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:36:40.661 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:40 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:40.326+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:36:40.661 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:40 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:40.326+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:36:40.982 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:40 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:40.681+0000 7f814fc80700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:05.126625+0000 front 2024-01-24T13:36:05.126588+0000 (oldest deadline 2024-01-24T13:36:29.826734+0000) 2024-01-24T13:36:40.982 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:40 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:40.681+0000 7f814fc80700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.628400+0000 front 2024-01-24T13:36:12.628490+0000 (oldest deadline 2024-01-24T13:36:38.528258+0000) 2024-01-24T13:36:41.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:40 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:40.661+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.895912+0000 front 2024-01-24T13:35:57.895940+0000 (oldest deadline 2024-01-24T13:36:21.995938+0000) 2024-01-24T13:36:41.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:40 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:40.661+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.197664+0000 front 2024-01-24T13:36:07.197789+0000 (oldest deadline 2024-01-24T13:36:31.897749+0000) 2024-01-24T13:36:41.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:40 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:40.661+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:13.598791+0000 front 2024-01-24T13:36:13.598854+0000 (oldest deadline 2024-01-24T13:36:36.499036+0000) 2024-01-24T13:36:41.099 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:40 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:40.795+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:58.893086+0000 front 2024-01-24T13:35:58.893135+0000 (oldest deadline 2024-01-24T13:36:24.693987+0000) 2024-01-24T13:36:41.099 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:40 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:40.795+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:04.694619+0000 front 2024-01-24T13:36:04.694633+0000 (oldest deadline 2024-01-24T13:36:29.394613+0000) 2024-01-24T13:36:41.099 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:40 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:40.795+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:13.396327+0000 front 2024-01-24T13:36:13.396184+0000 (oldest deadline 2024-01-24T13:36:36.896245+0000) 2024-01-24T13:36:41.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:41 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:41.034+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:36:41.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:41 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:41.034+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:36:41.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:41 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:41.034+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.121624+0000 front 2024-01-24T13:36:12.121460+0000 (oldest deadline 2024-01-24T13:36:37.321997+0000) 2024-01-24T13:36:41.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:41 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:41.102+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:36:41.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:41 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:41.103+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.684218+0000 front 2024-01-24T13:36:08.684295+0000 (oldest deadline 2024-01-24T13:36:32.784244+0000) 2024-01-24T13:36:41.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:41 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:41.103+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.784926+0000 front 2024-01-24T13:36:12.784800+0000 (oldest deadline 2024-01-24T13:36:38.684866+0000) 2024-01-24T13:36:41.612 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:41 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:41.318+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:36:41.612 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:41 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:41.318+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:36:41.612 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:41 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:41.318+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:36:41.982 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:41 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:41.636+0000 7f814fc80700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:05.126625+0000 front 2024-01-24T13:36:05.126588+0000 (oldest deadline 2024-01-24T13:36:29.826734+0000) 2024-01-24T13:36:41.982 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:41 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:41.636+0000 7f814fc80700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.628400+0000 front 2024-01-24T13:36:12.628490+0000 (oldest deadline 2024-01-24T13:36:38.528258+0000) 2024-01-24T13:36:42.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:41 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:41.612+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.895912+0000 front 2024-01-24T13:35:57.895940+0000 (oldest deadline 2024-01-24T13:36:21.995938+0000) 2024-01-24T13:36:42.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:41 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:41.612+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.197664+0000 front 2024-01-24T13:36:07.197789+0000 (oldest deadline 2024-01-24T13:36:31.897749+0000) 2024-01-24T13:36:42.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:41 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:41.612+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:13.598791+0000 front 2024-01-24T13:36:13.598854+0000 (oldest deadline 2024-01-24T13:36:36.499036+0000) 2024-01-24T13:36:42.094 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:41 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:41.823+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:58.893086+0000 front 2024-01-24T13:35:58.893135+0000 (oldest deadline 2024-01-24T13:36:24.693987+0000) 2024-01-24T13:36:42.094 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:41 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:41.823+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:04.694619+0000 front 2024-01-24T13:36:04.694633+0000 (oldest deadline 2024-01-24T13:36:29.394613+0000) 2024-01-24T13:36:42.094 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:41 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:41.823+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:13.396327+0000 front 2024-01-24T13:36:13.396184+0000 (oldest deadline 2024-01-24T13:36:36.896245+0000) 2024-01-24T13:36:42.349 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:42 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:42.093+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:36:42.349 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:42 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:42.093+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.684218+0000 front 2024-01-24T13:36:08.684295+0000 (oldest deadline 2024-01-24T13:36:32.784244+0000) 2024-01-24T13:36:42.349 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:42 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:42.093+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.784926+0000 front 2024-01-24T13:36:12.784800+0000 (oldest deadline 2024-01-24T13:36:38.684866+0000) 2024-01-24T13:36:42.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:42 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:42.044+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:36:42.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:42 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:42.044+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:36:42.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:42 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:42.044+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.121624+0000 front 2024-01-24T13:36:12.121460+0000 (oldest deadline 2024-01-24T13:36:37.321997+0000) 2024-01-24T13:36:42.633 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:42 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:42.309+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:36:42.633 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:42 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:42.309+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:36:42.633 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:42 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:42.309+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:36:42.981 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:42 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:42.588+0000 7f814fc80700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:05.126625+0000 front 2024-01-24T13:36:05.126588+0000 (oldest deadline 2024-01-24T13:36:29.826734+0000) 2024-01-24T13:36:42.982 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:42 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:42.588+0000 7f814fc80700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.628400+0000 front 2024-01-24T13:36:12.628490+0000 (oldest deadline 2024-01-24T13:36:38.528258+0000) 2024-01-24T13:36:43.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:42 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:42.632+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.895912+0000 front 2024-01-24T13:35:57.895940+0000 (oldest deadline 2024-01-24T13:36:21.995938+0000) 2024-01-24T13:36:43.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:42 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:42.632+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.197664+0000 front 2024-01-24T13:36:07.197789+0000 (oldest deadline 2024-01-24T13:36:31.897749+0000) 2024-01-24T13:36:43.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:42 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:42.632+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:13.598791+0000 front 2024-01-24T13:36:13.598854+0000 (oldest deadline 2024-01-24T13:36:36.499036+0000) 2024-01-24T13:36:43.077 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:42 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:42.815+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:58.893086+0000 front 2024-01-24T13:35:58.893135+0000 (oldest deadline 2024-01-24T13:36:24.693987+0000) 2024-01-24T13:36:43.077 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:42 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:42.815+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:04.694619+0000 front 2024-01-24T13:36:04.694633+0000 (oldest deadline 2024-01-24T13:36:29.394613+0000) 2024-01-24T13:36:43.077 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:42 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:42.815+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:13.396327+0000 front 2024-01-24T13:36:13.396184+0000 (oldest deadline 2024-01-24T13:36:36.896245+0000) 2024-01-24T13:36:43.349 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:43 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:43.076+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:36:43.349 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:43 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:43.076+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.684218+0000 front 2024-01-24T13:36:08.684295+0000 (oldest deadline 2024-01-24T13:36:32.784244+0000) 2024-01-24T13:36:43.349 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:43 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:43.076+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.784926+0000 front 2024-01-24T13:36:12.784800+0000 (oldest deadline 2024-01-24T13:36:38.684866+0000) 2024-01-24T13:36:43.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:43 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:43.028+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:36:43.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:43 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:43.028+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:36:43.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:43 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:43.028+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.121624+0000 front 2024-01-24T13:36:12.121460+0000 (oldest deadline 2024-01-24T13:36:37.321997+0000) 2024-01-24T13:36:43.642 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:43 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:43.314+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:36:43.642 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:43 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:43.314+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:36:43.642 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:43 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:43.314+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:36:43.982 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:43 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:43.618+0000 7f814fc80700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:05.126625+0000 front 2024-01-24T13:36:05.126588+0000 (oldest deadline 2024-01-24T13:36:29.826734+0000) 2024-01-24T13:36:43.982 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:43 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:43.618+0000 7f814fc80700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.628400+0000 front 2024-01-24T13:36:12.628490+0000 (oldest deadline 2024-01-24T13:36:38.528258+0000) 2024-01-24T13:36:44.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:43 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:43.641+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.895912+0000 front 2024-01-24T13:35:57.895940+0000 (oldest deadline 2024-01-24T13:36:21.995938+0000) 2024-01-24T13:36:44.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:43 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:43.641+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.197664+0000 front 2024-01-24T13:36:07.197789+0000 (oldest deadline 2024-01-24T13:36:31.897749+0000) 2024-01-24T13:36:44.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:43 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:43.641+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:13.598791+0000 front 2024-01-24T13:36:13.598854+0000 (oldest deadline 2024-01-24T13:36:36.499036+0000) 2024-01-24T13:36:44.095 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:43 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:43.802+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:58.893086+0000 front 2024-01-24T13:35:58.893135+0000 (oldest deadline 2024-01-24T13:36:24.693987+0000) 2024-01-24T13:36:44.095 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:43 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:43.802+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:04.694619+0000 front 2024-01-24T13:36:04.694633+0000 (oldest deadline 2024-01-24T13:36:29.394613+0000) 2024-01-24T13:36:44.095 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:43 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:43.802+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:13.396327+0000 front 2024-01-24T13:36:13.396184+0000 (oldest deadline 2024-01-24T13:36:36.896245+0000) 2024-01-24T13:36:44.095 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:43 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6[75791]: 2024-01-24T13:36:43.802+0000 7f8589944700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:21.399019+0000 front 2024-01-24T13:36:21.398980+0000 (oldest deadline 2024-01-24T13:36:43.699320+0000) 2024-01-24T13:36:44.349 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:44 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:44.095+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:36:44.349 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:44 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:44.095+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.684218+0000 front 2024-01-24T13:36:08.684295+0000 (oldest deadline 2024-01-24T13:36:32.784244+0000) 2024-01-24T13:36:44.349 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:44 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:44.095+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.784926+0000 front 2024-01-24T13:36:12.784800+0000 (oldest deadline 2024-01-24T13:36:38.684866+0000) 2024-01-24T13:36:44.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:44 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:44.006+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:36:44.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:44 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:44.006+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:36:44.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:44 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:44.006+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.121624+0000 front 2024-01-24T13:36:12.121460+0000 (oldest deadline 2024-01-24T13:36:37.321997+0000) 2024-01-24T13:36:44.554 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:44 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:44.301+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:36:44.554 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:44 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:44.301+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:36:44.555 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:44 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:44.301+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:36:44.982 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:44 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:44.624+0000 7f814fc80700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:05.126625+0000 front 2024-01-24T13:36:05.126588+0000 (oldest deadline 2024-01-24T13:36:29.826734+0000) 2024-01-24T13:36:44.982 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:44 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:44.624+0000 7f814fc80700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.628400+0000 front 2024-01-24T13:36:12.628490+0000 (oldest deadline 2024-01-24T13:36:38.528258+0000) 2024-01-24T13:36:45.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:44 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:44.659+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.895912+0000 front 2024-01-24T13:35:57.895940+0000 (oldest deadline 2024-01-24T13:36:21.995938+0000) 2024-01-24T13:36:45.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:44 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:44.659+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.197664+0000 front 2024-01-24T13:36:07.197789+0000 (oldest deadline 2024-01-24T13:36:31.897749+0000) 2024-01-24T13:36:45.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:44 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:44.659+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:13.598791+0000 front 2024-01-24T13:36:13.598854+0000 (oldest deadline 2024-01-24T13:36:36.499036+0000) 2024-01-24T13:36:45.349 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jan 24 13:36:45 smithi029 bash[97227]: ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-6 2024-01-24T13:36:45.349 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:45 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:45.135+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:36:45.349 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:45 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:45.135+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.684218+0000 front 2024-01-24T13:36:08.684295+0000 (oldest deadline 2024-01-24T13:36:32.784244+0000) 2024-01-24T13:36:45.349 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:45 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:45.135+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.784926+0000 front 2024-01-24T13:36:12.784800+0000 (oldest deadline 2024-01-24T13:36:38.684866+0000) 2024-01-24T13:36:45.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:45 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:45.015+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:36:45.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:45 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:45.015+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:36:45.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:45 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:45.015+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.121624+0000 front 2024-01-24T13:36:12.121460+0000 (oldest deadline 2024-01-24T13:36:37.321997+0000) 2024-01-24T13:36:45.554 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:45 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:45.286+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:36:45.554 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:45 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:45.286+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:36:45.554 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:45 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:45.286+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:36:45.982 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:45 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:45.626+0000 7f814fc80700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:05.126625+0000 front 2024-01-24T13:36:05.126588+0000 (oldest deadline 2024-01-24T13:36:29.826734+0000) 2024-01-24T13:36:45.982 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:45 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:45.626+0000 7f814fc80700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.628400+0000 front 2024-01-24T13:36:12.628490+0000 (oldest deadline 2024-01-24T13:36:38.528258+0000) 2024-01-24T13:36:46.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:45 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:45.704+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.895912+0000 front 2024-01-24T13:35:57.895940+0000 (oldest deadline 2024-01-24T13:36:21.995938+0000) 2024-01-24T13:36:46.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:45 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:45.704+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.197664+0000 front 2024-01-24T13:36:07.197789+0000 (oldest deadline 2024-01-24T13:36:31.897749+0000) 2024-01-24T13:36:46.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:45 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:45.704+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:13.598791+0000 front 2024-01-24T13:36:13.598854+0000 (oldest deadline 2024-01-24T13:36:36.499036+0000) 2024-01-24T13:36:46.436 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:46 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:46.125+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:36:46.437 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:46 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:46.125+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.684218+0000 front 2024-01-24T13:36:08.684295+0000 (oldest deadline 2024-01-24T13:36:32.784244+0000) 2024-01-24T13:36:46.437 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:46 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:46.125+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.784926+0000 front 2024-01-24T13:36:12.784800+0000 (oldest deadline 2024-01-24T13:36:38.684866+0000) 2024-01-24T13:36:46.446 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@osd.6.service' 2024-01-24T13:36:46.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:46 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:46.028+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:36:46.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:46 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:46.028+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:36:46.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:46 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:46.028+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.121624+0000 front 2024-01-24T13:36:12.121460+0000 (oldest deadline 2024-01-24T13:36:37.321997+0000) 2024-01-24T13:36:46.498 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T13:36:46.498 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-01-24T13:36:46.498 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-01-24T13:36:46.498 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-83afcf5e-babc-11ee-95b1-87774f69a715@osd.7 2024-01-24T13:36:46.743 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:46 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:46.318+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:36:46.743 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:46 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:46.319+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:36:46.744 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:46 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:46.319+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:36:46.982 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:46 smithi163 systemd[1]: Stopping Ceph osd.7 for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:36:46.983 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:46 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:46.636+0000 7f814fc80700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:05.126625+0000 front 2024-01-24T13:36:05.126588+0000 (oldest deadline 2024-01-24T13:36:29.826734+0000) 2024-01-24T13:36:46.983 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:46 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:46.636+0000 7f814fc80700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.628400+0000 front 2024-01-24T13:36:12.628490+0000 (oldest deadline 2024-01-24T13:36:38.528258+0000) 2024-01-24T13:36:46.983 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:46 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:46.749+0000 7f8154e8d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T13:36:46.983 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:46 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:46.749+0000 7f8154e8d700 -1 osd.7 85 *** Got signal Terminated *** 2024-01-24T13:36:46.983 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:46 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:46.749+0000 7f8154e8d700 -1 osd.7 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-24T13:36:47.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:46 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:46.743+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.895912+0000 front 2024-01-24T13:35:57.895940+0000 (oldest deadline 2024-01-24T13:36:21.995938+0000) 2024-01-24T13:36:47.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:46 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:46.743+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.197664+0000 front 2024-01-24T13:36:07.197789+0000 (oldest deadline 2024-01-24T13:36:31.897749+0000) 2024-01-24T13:36:47.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:46 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:46.743+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:13.598791+0000 front 2024-01-24T13:36:13.598854+0000 (oldest deadline 2024-01-24T13:36:36.499036+0000) 2024-01-24T13:36:47.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:47 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:47.009+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:36:47.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:47 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:47.009+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:36:47.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:47 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:47.009+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.121624+0000 front 2024-01-24T13:36:12.121460+0000 (oldest deadline 2024-01-24T13:36:37.321997+0000) 2024-01-24T13:36:47.554 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:47 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:47.288+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:36:47.555 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:47 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:47.288+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:36:47.555 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:47 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:47.288+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:36:47.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:47 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:47.102+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:36:47.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:47 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:47.102+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.684218+0000 front 2024-01-24T13:36:08.684295+0000 (oldest deadline 2024-01-24T13:36:32.784244+0000) 2024-01-24T13:36:47.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:47 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:47.102+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.784926+0000 front 2024-01-24T13:36:12.784800+0000 (oldest deadline 2024-01-24T13:36:38.684866+0000) 2024-01-24T13:36:47.982 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:47 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:47.605+0000 7f814fc80700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:05.126625+0000 front 2024-01-24T13:36:05.126588+0000 (oldest deadline 2024-01-24T13:36:29.826734+0000) 2024-01-24T13:36:47.982 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:47 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:47.605+0000 7f814fc80700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.628400+0000 front 2024-01-24T13:36:12.628490+0000 (oldest deadline 2024-01-24T13:36:38.528258+0000) 2024-01-24T13:36:48.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:47 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:47.725+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.895912+0000 front 2024-01-24T13:35:57.895940+0000 (oldest deadline 2024-01-24T13:36:21.995938+0000) 2024-01-24T13:36:48.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:47 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:47.725+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.197664+0000 front 2024-01-24T13:36:07.197789+0000 (oldest deadline 2024-01-24T13:36:31.897749+0000) 2024-01-24T13:36:48.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:47 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:47.725+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:13.598791+0000 front 2024-01-24T13:36:13.598854+0000 (oldest deadline 2024-01-24T13:36:36.499036+0000) 2024-01-24T13:36:48.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:48 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:48.009+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:36:48.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:48 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:48.009+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:36:48.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:48 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:48.009+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.121624+0000 front 2024-01-24T13:36:12.121460+0000 (oldest deadline 2024-01-24T13:36:37.321997+0000) 2024-01-24T13:36:48.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:48 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:48.131+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:36:48.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:48 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:48.131+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.684218+0000 front 2024-01-24T13:36:08.684295+0000 (oldest deadline 2024-01-24T13:36:32.784244+0000) 2024-01-24T13:36:48.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:48 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:48.131+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.784926+0000 front 2024-01-24T13:36:12.784800+0000 (oldest deadline 2024-01-24T13:36:38.684866+0000) 2024-01-24T13:36:48.712 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:48 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:48.333+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:36:48.712 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:48 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:48.333+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:36:48.712 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:48 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:48.333+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:36:48.982 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:48 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:48.573+0000 7f814fc80700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:05.126625+0000 front 2024-01-24T13:36:05.126588+0000 (oldest deadline 2024-01-24T13:36:29.826734+0000) 2024-01-24T13:36:48.982 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:48 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:48.573+0000 7f814fc80700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.628400+0000 front 2024-01-24T13:36:12.628490+0000 (oldest deadline 2024-01-24T13:36:38.528258+0000) 2024-01-24T13:36:48.982 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:48 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:48.573+0000 7f814fc80700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:22.629389+0000 front 2024-01-24T13:36:22.629263+0000 (oldest deadline 2024-01-24T13:36:48.529199+0000) 2024-01-24T13:36:49.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:48 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:48.711+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.895912+0000 front 2024-01-24T13:35:57.895940+0000 (oldest deadline 2024-01-24T13:36:21.995938+0000) 2024-01-24T13:36:49.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:48 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:48.711+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.197664+0000 front 2024-01-24T13:36:07.197789+0000 (oldest deadline 2024-01-24T13:36:31.897749+0000) 2024-01-24T13:36:49.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:48 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:48.711+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:13.598791+0000 front 2024-01-24T13:36:13.598854+0000 (oldest deadline 2024-01-24T13:36:36.499036+0000) 2024-01-24T13:36:49.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:48 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:48.711+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:22.400439+0000 front 2024-01-24T13:36:22.400454+0000 (oldest deadline 2024-01-24T13:36:48.300412+0000) 2024-01-24T13:36:49.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:49 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:49.005+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:36:49.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:49 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:49.005+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:36:49.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:49 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:49.005+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.121624+0000 front 2024-01-24T13:36:12.121460+0000 (oldest deadline 2024-01-24T13:36:37.321997+0000) 2024-01-24T13:36:49.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:49 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:49.005+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:22.623093+0000 front 2024-01-24T13:36:22.623173+0000 (oldest deadline 2024-01-24T13:36:48.523114+0000) 2024-01-24T13:36:49.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:49 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:49.145+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:36:49.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:49 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:49.145+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.684218+0000 front 2024-01-24T13:36:08.684295+0000 (oldest deadline 2024-01-24T13:36:32.784244+0000) 2024-01-24T13:36:49.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:49 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:49.145+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.784926+0000 front 2024-01-24T13:36:12.784800+0000 (oldest deadline 2024-01-24T13:36:38.684866+0000) 2024-01-24T13:36:49.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:49 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:49.145+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:18.685211+0000 front 2024-01-24T13:36:18.685291+0000 (oldest deadline 2024-01-24T13:36:48.685987+0000) 2024-01-24T13:36:49.735 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:49 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:49.361+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:36:49.735 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:49 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:49.361+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:36:49.735 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:49 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:49.361+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:36:49.982 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:49 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:49.599+0000 7f814fc80700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:05.126625+0000 front 2024-01-24T13:36:05.126588+0000 (oldest deadline 2024-01-24T13:36:29.826734+0000) 2024-01-24T13:36:49.982 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:49 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:49.599+0000 7f814fc80700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.628400+0000 front 2024-01-24T13:36:12.628490+0000 (oldest deadline 2024-01-24T13:36:38.528258+0000) 2024-01-24T13:36:49.982 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:49 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:49.599+0000 7f814fc80700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:22.629389+0000 front 2024-01-24T13:36:22.629263+0000 (oldest deadline 2024-01-24T13:36:48.529199+0000) 2024-01-24T13:36:50.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:49 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:49.734+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.895912+0000 front 2024-01-24T13:35:57.895940+0000 (oldest deadline 2024-01-24T13:36:21.995938+0000) 2024-01-24T13:36:50.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:49 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:49.734+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.197664+0000 front 2024-01-24T13:36:07.197789+0000 (oldest deadline 2024-01-24T13:36:31.897749+0000) 2024-01-24T13:36:50.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:49 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:49.734+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:13.598791+0000 front 2024-01-24T13:36:13.598854+0000 (oldest deadline 2024-01-24T13:36:36.499036+0000) 2024-01-24T13:36:50.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:49 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:49.734+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:22.400439+0000 front 2024-01-24T13:36:22.400454+0000 (oldest deadline 2024-01-24T13:36:48.300412+0000) 2024-01-24T13:36:50.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:50 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:50.020+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:36:50.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:50 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:50.020+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:36:50.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:50 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:50.020+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.121624+0000 front 2024-01-24T13:36:12.121460+0000 (oldest deadline 2024-01-24T13:36:37.321997+0000) 2024-01-24T13:36:50.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:50 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:50.020+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:22.623093+0000 front 2024-01-24T13:36:22.623173+0000 (oldest deadline 2024-01-24T13:36:48.523114+0000) 2024-01-24T13:36:50.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:50 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:50.158+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:36:50.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:50 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:50.158+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.684218+0000 front 2024-01-24T13:36:08.684295+0000 (oldest deadline 2024-01-24T13:36:32.784244+0000) 2024-01-24T13:36:50.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:50 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:50.158+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.784926+0000 front 2024-01-24T13:36:12.784800+0000 (oldest deadline 2024-01-24T13:36:38.684866+0000) 2024-01-24T13:36:50.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:50 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:50.158+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:18.685211+0000 front 2024-01-24T13:36:18.685291+0000 (oldest deadline 2024-01-24T13:36:48.685987+0000) 2024-01-24T13:36:50.763 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:50 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:50.382+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:36:50.763 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:50 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:50.382+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:36:50.763 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:50 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:50.382+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:36:50.982 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:50 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:50.567+0000 7f814fc80700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:05.126625+0000 front 2024-01-24T13:36:05.126588+0000 (oldest deadline 2024-01-24T13:36:29.826734+0000) 2024-01-24T13:36:50.982 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:50 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:50.567+0000 7f814fc80700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.628400+0000 front 2024-01-24T13:36:12.628490+0000 (oldest deadline 2024-01-24T13:36:38.528258+0000) 2024-01-24T13:36:50.982 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:50 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:50.567+0000 7f814fc80700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:22.629389+0000 front 2024-01-24T13:36:22.629263+0000 (oldest deadline 2024-01-24T13:36:48.529199+0000) 2024-01-24T13:36:51.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:50 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:50.762+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.895912+0000 front 2024-01-24T13:35:57.895940+0000 (oldest deadline 2024-01-24T13:36:21.995938+0000) 2024-01-24T13:36:51.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:50 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:50.762+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.197664+0000 front 2024-01-24T13:36:07.197789+0000 (oldest deadline 2024-01-24T13:36:31.897749+0000) 2024-01-24T13:36:51.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:50 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:50.762+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:13.598791+0000 front 2024-01-24T13:36:13.598854+0000 (oldest deadline 2024-01-24T13:36:36.499036+0000) 2024-01-24T13:36:51.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:50 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:50.762+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:22.400439+0000 front 2024-01-24T13:36:22.400454+0000 (oldest deadline 2024-01-24T13:36:48.300412+0000) 2024-01-24T13:36:51.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:51 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:51.049+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:36:51.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:51 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:51.049+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:36:51.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:51 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:51.049+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.121624+0000 front 2024-01-24T13:36:12.121460+0000 (oldest deadline 2024-01-24T13:36:37.321997+0000) 2024-01-24T13:36:51.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:51 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:51.049+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:22.623093+0000 front 2024-01-24T13:36:22.623173+0000 (oldest deadline 2024-01-24T13:36:48.523114+0000) 2024-01-24T13:36:51.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:51 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:51.139+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:36:51.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:51 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:51.139+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.684218+0000 front 2024-01-24T13:36:08.684295+0000 (oldest deadline 2024-01-24T13:36:32.784244+0000) 2024-01-24T13:36:51.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:51 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:51.139+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.784926+0000 front 2024-01-24T13:36:12.784800+0000 (oldest deadline 2024-01-24T13:36:38.684866+0000) 2024-01-24T13:36:51.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:51 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:51.139+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:18.685211+0000 front 2024-01-24T13:36:18.685291+0000 (oldest deadline 2024-01-24T13:36:48.685987+0000) 2024-01-24T13:36:51.794 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:51 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:51.342+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:36:51.794 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:51 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:51.342+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:36:51.794 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:51 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:51.342+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:36:51.982 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:51 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:51.565+0000 7f814fc80700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:05.126625+0000 front 2024-01-24T13:36:05.126588+0000 (oldest deadline 2024-01-24T13:36:29.826734+0000) 2024-01-24T13:36:51.982 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:51 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:51.565+0000 7f814fc80700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.628400+0000 front 2024-01-24T13:36:12.628490+0000 (oldest deadline 2024-01-24T13:36:38.528258+0000) 2024-01-24T13:36:51.982 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:51 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7[67304]: 2024-01-24T13:36:51.565+0000 7f814fc80700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:22.629389+0000 front 2024-01-24T13:36:22.629263+0000 (oldest deadline 2024-01-24T13:36:48.529199+0000) 2024-01-24T13:36:52.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:51 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:51.794+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.895912+0000 front 2024-01-24T13:35:57.895940+0000 (oldest deadline 2024-01-24T13:36:21.995938+0000) 2024-01-24T13:36:52.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:51 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:51.794+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.197664+0000 front 2024-01-24T13:36:07.197789+0000 (oldest deadline 2024-01-24T13:36:31.897749+0000) 2024-01-24T13:36:52.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:51 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:51.794+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:13.598791+0000 front 2024-01-24T13:36:13.598854+0000 (oldest deadline 2024-01-24T13:36:36.499036+0000) 2024-01-24T13:36:52.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:51 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:51.794+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:22.400439+0000 front 2024-01-24T13:36:22.400454+0000 (oldest deadline 2024-01-24T13:36:48.300412+0000) 2024-01-24T13:36:52.415 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:52 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:52.072+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:36:52.415 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:52 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:52.072+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:36:52.415 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:52 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:52.072+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.121624+0000 front 2024-01-24T13:36:12.121460+0000 (oldest deadline 2024-01-24T13:36:37.321997+0000) 2024-01-24T13:36:52.415 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:52 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:52.072+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:22.623093+0000 front 2024-01-24T13:36:22.623173+0000 (oldest deadline 2024-01-24T13:36:48.523114+0000) 2024-01-24T13:36:52.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:52 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:52.150+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:36:52.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:52 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:52.150+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.684218+0000 front 2024-01-24T13:36:08.684295+0000 (oldest deadline 2024-01-24T13:36:32.784244+0000) 2024-01-24T13:36:52.600 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:52 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:52.150+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.784926+0000 front 2024-01-24T13:36:12.784800+0000 (oldest deadline 2024-01-24T13:36:38.684866+0000) 2024-01-24T13:36:52.600 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:52 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:52.150+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:18.685211+0000 front 2024-01-24T13:36:18.685291+0000 (oldest deadline 2024-01-24T13:36:48.685987+0000) 2024-01-24T13:36:52.600 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:52 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:52.150+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:29.786971+0000 front 2024-01-24T13:36:29.787029+0000 (oldest deadline 2024-01-24T13:36:51.987063+0000) 2024-01-24T13:36:52.732 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jan 24 13:36:52 smithi163 bash[85094]: ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-7 2024-01-24T13:36:52.783 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:52 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:52.341+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:36:52.783 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:52 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:52.341+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:36:52.783 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:52 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:52.341+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:36:52.784 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:52 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:52.341+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:27.142218+0000 front 2024-01-24T13:36:27.142234+0000 (oldest deadline 2024-01-24T13:36:51.842262+0000) 2024-01-24T13:36:53.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:52 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:52.781+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.895912+0000 front 2024-01-24T13:35:57.895940+0000 (oldest deadline 2024-01-24T13:36:21.995938+0000) 2024-01-24T13:36:53.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:52 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:52.781+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.197664+0000 front 2024-01-24T13:36:07.197789+0000 (oldest deadline 2024-01-24T13:36:31.897749+0000) 2024-01-24T13:36:53.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:52 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:52.781+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:13.598791+0000 front 2024-01-24T13:36:13.598854+0000 (oldest deadline 2024-01-24T13:36:36.499036+0000) 2024-01-24T13:36:53.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:52 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:52.781+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:22.400439+0000 front 2024-01-24T13:36:22.400454+0000 (oldest deadline 2024-01-24T13:36:48.300412+0000) 2024-01-24T13:36:53.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:53 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:53.033+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:36:53.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:53 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:53.033+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:36:53.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:53 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:53.033+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.121624+0000 front 2024-01-24T13:36:12.121460+0000 (oldest deadline 2024-01-24T13:36:37.321997+0000) 2024-01-24T13:36:53.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:53 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:53.033+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:22.623093+0000 front 2024-01-24T13:36:22.623173+0000 (oldest deadline 2024-01-24T13:36:48.523114+0000) 2024-01-24T13:36:53.600 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:53 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:53.102+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:36:53.600 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:53 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:53.102+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.684218+0000 front 2024-01-24T13:36:08.684295+0000 (oldest deadline 2024-01-24T13:36:32.784244+0000) 2024-01-24T13:36:53.600 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:53 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:53.102+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.784926+0000 front 2024-01-24T13:36:12.784800+0000 (oldest deadline 2024-01-24T13:36:38.684866+0000) 2024-01-24T13:36:53.600 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:53 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:53.102+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:18.685211+0000 front 2024-01-24T13:36:18.685291+0000 (oldest deadline 2024-01-24T13:36:48.685987+0000) 2024-01-24T13:36:53.600 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:53 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:53.102+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:29.786971+0000 front 2024-01-24T13:36:29.787029+0000 (oldest deadline 2024-01-24T13:36:51.987063+0000) 2024-01-24T13:36:53.669 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@osd.7.service' 2024-01-24T13:36:53.710 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T13:36:53.710 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-01-24T13:36:53.710 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-01-24T13:36:53.710 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-83afcf5e-babc-11ee-95b1-87774f69a715@osd.8 2024-01-24T13:36:53.740 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:53 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:53.319+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:36:53.740 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:53 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:53.319+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:36:53.741 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:53 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:53.319+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:36:53.741 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:53 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:53.319+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:27.142218+0000 front 2024-01-24T13:36:27.142234+0000 (oldest deadline 2024-01-24T13:36:51.842262+0000) 2024-01-24T13:36:54.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:53 smithi187 systemd[1]: Stopping Ceph osd.8 for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:36:54.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:53 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:53.774+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.895912+0000 front 2024-01-24T13:35:57.895940+0000 (oldest deadline 2024-01-24T13:36:21.995938+0000) 2024-01-24T13:36:54.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:53 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:53.774+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.197664+0000 front 2024-01-24T13:36:07.197789+0000 (oldest deadline 2024-01-24T13:36:31.897749+0000) 2024-01-24T13:36:54.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:53 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:53.774+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:13.598791+0000 front 2024-01-24T13:36:13.598854+0000 (oldest deadline 2024-01-24T13:36:36.499036+0000) 2024-01-24T13:36:54.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:53 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:53.774+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:22.400439+0000 front 2024-01-24T13:36:22.400454+0000 (oldest deadline 2024-01-24T13:36:48.300412+0000) 2024-01-24T13:36:54.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:53 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:53.774+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:28.300976+0000 front 2024-01-24T13:36:28.301032+0000 (oldest deadline 2024-01-24T13:36:53.000952+0000) 2024-01-24T13:36:54.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:53 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:53.942+0000 7f1bf2ac0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T13:36:54.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:53 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:53.942+0000 7f1bf2ac0700 -1 osd.8 85 *** Got signal Terminated *** 2024-01-24T13:36:54.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:53 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:53.942+0000 7f1bf2ac0700 -1 osd.8 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-24T13:36:54.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:53 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:53.984+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:36:54.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:53 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:53.984+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:36:54.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:53 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:53.984+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.121624+0000 front 2024-01-24T13:36:12.121460+0000 (oldest deadline 2024-01-24T13:36:37.321997+0000) 2024-01-24T13:36:54.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:53 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:53.984+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:22.623093+0000 front 2024-01-24T13:36:22.623173+0000 (oldest deadline 2024-01-24T13:36:48.523114+0000) 2024-01-24T13:36:54.554 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:54 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:54.300+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:36:54.554 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:54 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:54.300+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:36:54.554 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:54 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:54.300+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:36:54.555 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:54 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:54.300+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:27.142218+0000 front 2024-01-24T13:36:27.142234+0000 (oldest deadline 2024-01-24T13:36:51.842262+0000) 2024-01-24T13:36:54.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:54 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:54.120+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:36:54.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:54 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:54.120+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.684218+0000 front 2024-01-24T13:36:08.684295+0000 (oldest deadline 2024-01-24T13:36:32.784244+0000) 2024-01-24T13:36:54.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:54 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:54.120+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.784926+0000 front 2024-01-24T13:36:12.784800+0000 (oldest deadline 2024-01-24T13:36:38.684866+0000) 2024-01-24T13:36:54.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:54 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:54.120+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:18.685211+0000 front 2024-01-24T13:36:18.685291+0000 (oldest deadline 2024-01-24T13:36:48.685987+0000) 2024-01-24T13:36:54.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:54 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:54.120+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:29.786971+0000 front 2024-01-24T13:36:29.787029+0000 (oldest deadline 2024-01-24T13:36:51.987063+0000) 2024-01-24T13:36:55.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:54 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:54.754+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.895912+0000 front 2024-01-24T13:35:57.895940+0000 (oldest deadline 2024-01-24T13:36:21.995938+0000) 2024-01-24T13:36:55.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:54 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:54.754+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.197664+0000 front 2024-01-24T13:36:07.197789+0000 (oldest deadline 2024-01-24T13:36:31.897749+0000) 2024-01-24T13:36:55.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:54 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:54.754+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:13.598791+0000 front 2024-01-24T13:36:13.598854+0000 (oldest deadline 2024-01-24T13:36:36.499036+0000) 2024-01-24T13:36:55.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:54 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:54.754+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:22.400439+0000 front 2024-01-24T13:36:22.400454+0000 (oldest deadline 2024-01-24T13:36:48.300412+0000) 2024-01-24T13:36:55.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:54 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:54.754+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:28.300976+0000 front 2024-01-24T13:36:28.301032+0000 (oldest deadline 2024-01-24T13:36:53.000952+0000) 2024-01-24T13:36:55.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:54 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:54.955+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:36:55.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:54 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:54.955+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:36:55.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:54 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:54.955+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.121624+0000 front 2024-01-24T13:36:12.121460+0000 (oldest deadline 2024-01-24T13:36:37.321997+0000) 2024-01-24T13:36:55.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:54 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:54.955+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:22.623093+0000 front 2024-01-24T13:36:22.623173+0000 (oldest deadline 2024-01-24T13:36:48.523114+0000) 2024-01-24T13:36:55.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:54 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:54.955+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:28.523591+0000 front 2024-01-24T13:36:28.523637+0000 (oldest deadline 2024-01-24T13:36:54.423692+0000) 2024-01-24T13:36:55.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:55 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:55.159+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:36:55.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:55 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:55.159+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.684218+0000 front 2024-01-24T13:36:08.684295+0000 (oldest deadline 2024-01-24T13:36:32.784244+0000) 2024-01-24T13:36:55.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:55 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:55.159+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.784926+0000 front 2024-01-24T13:36:12.784800+0000 (oldest deadline 2024-01-24T13:36:38.684866+0000) 2024-01-24T13:36:55.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:55 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:55.159+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:18.685211+0000 front 2024-01-24T13:36:18.685291+0000 (oldest deadline 2024-01-24T13:36:48.685987+0000) 2024-01-24T13:36:55.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:55 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:55.159+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:29.786971+0000 front 2024-01-24T13:36:29.787029+0000 (oldest deadline 2024-01-24T13:36:51.987063+0000) 2024-01-24T13:36:55.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:55 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:55.304+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:36:55.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:55 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:55.304+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:36:55.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:55 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:55.304+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:36:55.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:55 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:55.304+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:27.142218+0000 front 2024-01-24T13:36:27.142234+0000 (oldest deadline 2024-01-24T13:36:51.842262+0000) 2024-01-24T13:36:56.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:55 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:55.803+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.895912+0000 front 2024-01-24T13:35:57.895940+0000 (oldest deadline 2024-01-24T13:36:21.995938+0000) 2024-01-24T13:36:56.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:55 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:55.803+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.197664+0000 front 2024-01-24T13:36:07.197789+0000 (oldest deadline 2024-01-24T13:36:31.897749+0000) 2024-01-24T13:36:56.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:55 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:55.803+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:13.598791+0000 front 2024-01-24T13:36:13.598854+0000 (oldest deadline 2024-01-24T13:36:36.499036+0000) 2024-01-24T13:36:56.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:55 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:55.803+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:22.400439+0000 front 2024-01-24T13:36:22.400454+0000 (oldest deadline 2024-01-24T13:36:48.300412+0000) 2024-01-24T13:36:56.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:55 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:55.803+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:28.300976+0000 front 2024-01-24T13:36:28.301032+0000 (oldest deadline 2024-01-24T13:36:53.000952+0000) 2024-01-24T13:36:56.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:55 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:55.912+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:36:56.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:55 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:55.912+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:36:56.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:55 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:55.912+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.121624+0000 front 2024-01-24T13:36:12.121460+0000 (oldest deadline 2024-01-24T13:36:37.321997+0000) 2024-01-24T13:36:56.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:55 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:55.912+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:22.623093+0000 front 2024-01-24T13:36:22.623173+0000 (oldest deadline 2024-01-24T13:36:48.523114+0000) 2024-01-24T13:36:56.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:55 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:55.912+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:28.523591+0000 front 2024-01-24T13:36:28.523637+0000 (oldest deadline 2024-01-24T13:36:54.423692+0000) 2024-01-24T13:36:56.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:56 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:56.121+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:36:56.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:56 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:56.121+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.684218+0000 front 2024-01-24T13:36:08.684295+0000 (oldest deadline 2024-01-24T13:36:32.784244+0000) 2024-01-24T13:36:56.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:56 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:56.121+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.784926+0000 front 2024-01-24T13:36:12.784800+0000 (oldest deadline 2024-01-24T13:36:38.684866+0000) 2024-01-24T13:36:56.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:56 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:56.121+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:18.685211+0000 front 2024-01-24T13:36:18.685291+0000 (oldest deadline 2024-01-24T13:36:48.685987+0000) 2024-01-24T13:36:56.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:56 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:56.121+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:29.786971+0000 front 2024-01-24T13:36:29.787029+0000 (oldest deadline 2024-01-24T13:36:51.987063+0000) 2024-01-24T13:36:56.785 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:56 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:56.314+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:36:56.785 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:56 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:56.314+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:36:56.785 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:56 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:56.314+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:36:56.785 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:56 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:56.314+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:27.142218+0000 front 2024-01-24T13:36:27.142234+0000 (oldest deadline 2024-01-24T13:36:51.842262+0000) 2024-01-24T13:36:57.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:56 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:56.784+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.895912+0000 front 2024-01-24T13:35:57.895940+0000 (oldest deadline 2024-01-24T13:36:21.995938+0000) 2024-01-24T13:36:57.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:56 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:56.784+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.197664+0000 front 2024-01-24T13:36:07.197789+0000 (oldest deadline 2024-01-24T13:36:31.897749+0000) 2024-01-24T13:36:57.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:56 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:56.784+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:13.598791+0000 front 2024-01-24T13:36:13.598854+0000 (oldest deadline 2024-01-24T13:36:36.499036+0000) 2024-01-24T13:36:57.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:56 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:56.784+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:22.400439+0000 front 2024-01-24T13:36:22.400454+0000 (oldest deadline 2024-01-24T13:36:48.300412+0000) 2024-01-24T13:36:57.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:56 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:56.784+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:28.300976+0000 front 2024-01-24T13:36:28.301032+0000 (oldest deadline 2024-01-24T13:36:53.000952+0000) 2024-01-24T13:36:57.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:56 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:56.892+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:36:57.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:56 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:56.892+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:36:57.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:56 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:56.892+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.121624+0000 front 2024-01-24T13:36:12.121460+0000 (oldest deadline 2024-01-24T13:36:37.321997+0000) 2024-01-24T13:36:57.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:56 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:56.892+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:22.623093+0000 front 2024-01-24T13:36:22.623173+0000 (oldest deadline 2024-01-24T13:36:48.523114+0000) 2024-01-24T13:36:57.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:56 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:56.892+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:28.523591+0000 front 2024-01-24T13:36:28.523637+0000 (oldest deadline 2024-01-24T13:36:54.423692+0000) 2024-01-24T13:36:57.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:57 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:57.171+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:36:57.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:57 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:57.171+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.684218+0000 front 2024-01-24T13:36:08.684295+0000 (oldest deadline 2024-01-24T13:36:32.784244+0000) 2024-01-24T13:36:57.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:57 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:57.171+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.784926+0000 front 2024-01-24T13:36:12.784800+0000 (oldest deadline 2024-01-24T13:36:38.684866+0000) 2024-01-24T13:36:57.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:57 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:57.171+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:18.685211+0000 front 2024-01-24T13:36:18.685291+0000 (oldest deadline 2024-01-24T13:36:48.685987+0000) 2024-01-24T13:36:57.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:57 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:57.171+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:29.786971+0000 front 2024-01-24T13:36:29.787029+0000 (oldest deadline 2024-01-24T13:36:51.987063+0000) 2024-01-24T13:36:57.770 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:57 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:57.305+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:36:57.770 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:57 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:57.305+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:36:57.771 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:57 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:57.305+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:36:57.771 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:57 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:57.305+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:27.142218+0000 front 2024-01-24T13:36:27.142234+0000 (oldest deadline 2024-01-24T13:36:51.842262+0000) 2024-01-24T13:36:58.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:57 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:57.770+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.895912+0000 front 2024-01-24T13:35:57.895940+0000 (oldest deadline 2024-01-24T13:36:21.995938+0000) 2024-01-24T13:36:58.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:57 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:57.770+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.197664+0000 front 2024-01-24T13:36:07.197789+0000 (oldest deadline 2024-01-24T13:36:31.897749+0000) 2024-01-24T13:36:58.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:57 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:57.770+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:13.598791+0000 front 2024-01-24T13:36:13.598854+0000 (oldest deadline 2024-01-24T13:36:36.499036+0000) 2024-01-24T13:36:58.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:57 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:57.770+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:22.400439+0000 front 2024-01-24T13:36:22.400454+0000 (oldest deadline 2024-01-24T13:36:48.300412+0000) 2024-01-24T13:36:58.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:57 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:57.770+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:28.300976+0000 front 2024-01-24T13:36:28.301032+0000 (oldest deadline 2024-01-24T13:36:53.000952+0000) 2024-01-24T13:36:58.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:57 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:57.910+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:36:58.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:57 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:57.910+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:36:58.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:57 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:57.910+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.121624+0000 front 2024-01-24T13:36:12.121460+0000 (oldest deadline 2024-01-24T13:36:37.321997+0000) 2024-01-24T13:36:58.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:57 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:57.910+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:22.623093+0000 front 2024-01-24T13:36:22.623173+0000 (oldest deadline 2024-01-24T13:36:48.523114+0000) 2024-01-24T13:36:58.233 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:57 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:57.910+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:28.523591+0000 front 2024-01-24T13:36:28.523637+0000 (oldest deadline 2024-01-24T13:36:54.423692+0000) 2024-01-24T13:36:58.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:58 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:58.204+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:36:58.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:58 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:58.204+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.684218+0000 front 2024-01-24T13:36:08.684295+0000 (oldest deadline 2024-01-24T13:36:32.784244+0000) 2024-01-24T13:36:58.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:58 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:58.204+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.784926+0000 front 2024-01-24T13:36:12.784800+0000 (oldest deadline 2024-01-24T13:36:38.684866+0000) 2024-01-24T13:36:58.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:58 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:58.204+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:18.685211+0000 front 2024-01-24T13:36:18.685291+0000 (oldest deadline 2024-01-24T13:36:48.685987+0000) 2024-01-24T13:36:58.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:58 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:58.204+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:29.786971+0000 front 2024-01-24T13:36:29.787029+0000 (oldest deadline 2024-01-24T13:36:51.987063+0000) 2024-01-24T13:36:58.783 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:58 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:58.311+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:36:58.783 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:58 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:58.311+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:36:58.783 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:58 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:58.311+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:36:58.783 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:58 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:58.311+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:27.142218+0000 front 2024-01-24T13:36:27.142234+0000 (oldest deadline 2024-01-24T13:36:51.842262+0000) 2024-01-24T13:36:59.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:58 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:58.782+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:57.895912+0000 front 2024-01-24T13:35:57.895940+0000 (oldest deadline 2024-01-24T13:36:21.995938+0000) 2024-01-24T13:36:59.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:58 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:58.782+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.197664+0000 front 2024-01-24T13:36:07.197789+0000 (oldest deadline 2024-01-24T13:36:31.897749+0000) 2024-01-24T13:36:59.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:58 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:58.782+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:13.598791+0000 front 2024-01-24T13:36:13.598854+0000 (oldest deadline 2024-01-24T13:36:36.499036+0000) 2024-01-24T13:36:59.054 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:58 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:58.782+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:22.400439+0000 front 2024-01-24T13:36:22.400454+0000 (oldest deadline 2024-01-24T13:36:48.300412+0000) 2024-01-24T13:36:59.055 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:58 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8[66556]: 2024-01-24T13:36:58.782+0000 7f1bed0b2700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:28.300976+0000 front 2024-01-24T13:36:28.301032+0000 (oldest deadline 2024-01-24T13:36:53.000952+0000) 2024-01-24T13:36:59.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:58 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:58.895+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:36:59.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:58 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:58.895+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:36:59.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:58 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:58.895+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.121624+0000 front 2024-01-24T13:36:12.121460+0000 (oldest deadline 2024-01-24T13:36:37.321997+0000) 2024-01-24T13:36:59.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:58 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:58.895+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:22.623093+0000 front 2024-01-24T13:36:22.623173+0000 (oldest deadline 2024-01-24T13:36:48.523114+0000) 2024-01-24T13:36:59.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:58 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:58.895+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:28.523591+0000 front 2024-01-24T13:36:28.523637+0000 (oldest deadline 2024-01-24T13:36:54.423692+0000) 2024-01-24T13:36:59.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:58 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:58.895+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2024-01-24T13:36:34.424128+0000 front 2024-01-24T13:36:34.424006+0000 (oldest deadline 2024-01-24T13:36:58.524025+0000) 2024-01-24T13:36:59.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:59 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:59.198+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:36:59.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:59 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:59.198+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.684218+0000 front 2024-01-24T13:36:08.684295+0000 (oldest deadline 2024-01-24T13:36:32.784244+0000) 2024-01-24T13:36:59.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:59 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:59.198+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.784926+0000 front 2024-01-24T13:36:12.784800+0000 (oldest deadline 2024-01-24T13:36:38.684866+0000) 2024-01-24T13:36:59.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:59 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:59.198+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:18.685211+0000 front 2024-01-24T13:36:18.685291+0000 (oldest deadline 2024-01-24T13:36:48.685987+0000) 2024-01-24T13:36:59.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:36:59 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:36:59.198+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:29.786971+0000 front 2024-01-24T13:36:29.787029+0000 (oldest deadline 2024-01-24T13:36:51.987063+0000) 2024-01-24T13:36:59.804 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jan 24 13:36:59 smithi187 bash[83922]: ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-8 2024-01-24T13:36:59.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:59 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:59.358+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:36:59.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:59 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:59.358+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:36:59.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:59 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:59.358+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:36:59.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:36:59 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:36:59.358+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:27.142218+0000 front 2024-01-24T13:36:27.142234+0000 (oldest deadline 2024-01-24T13:36:51.842262+0000) 2024-01-24T13:37:00.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:59 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:59.918+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:37:00.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:59 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:59.919+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:37:00.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:59 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:59.919+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.121624+0000 front 2024-01-24T13:36:12.121460+0000 (oldest deadline 2024-01-24T13:36:37.321997+0000) 2024-01-24T13:37:00.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:59 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:59.919+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:22.623093+0000 front 2024-01-24T13:36:22.623173+0000 (oldest deadline 2024-01-24T13:36:48.523114+0000) 2024-01-24T13:37:00.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:59 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:59.919+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:28.523591+0000 front 2024-01-24T13:36:28.523637+0000 (oldest deadline 2024-01-24T13:36:54.423692+0000) 2024-01-24T13:37:00.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:36:59 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:36:59.919+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2024-01-24T13:36:34.424128+0000 front 2024-01-24T13:36:34.424006+0000 (oldest deadline 2024-01-24T13:36:58.524025+0000) 2024-01-24T13:37:00.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:00 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:37:00.236+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:37:00.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:00 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:37:00.236+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.684218+0000 front 2024-01-24T13:36:08.684295+0000 (oldest deadline 2024-01-24T13:36:32.784244+0000) 2024-01-24T13:37:00.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:00 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:37:00.236+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.784926+0000 front 2024-01-24T13:36:12.784800+0000 (oldest deadline 2024-01-24T13:36:38.684866+0000) 2024-01-24T13:37:00.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:00 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:37:00.236+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:18.685211+0000 front 2024-01-24T13:36:18.685291+0000 (oldest deadline 2024-01-24T13:36:48.685987+0000) 2024-01-24T13:37:00.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:00 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:37:00.236+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:29.786971+0000 front 2024-01-24T13:36:29.787029+0000 (oldest deadline 2024-01-24T13:36:51.987063+0000) 2024-01-24T13:37:00.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:00 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:37:00.236+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2024-01-24T13:36:37.188589+0000 front 2024-01-24T13:36:37.188497+0000 (oldest deadline 2024-01-24T13:37:00.088406+0000) 2024-01-24T13:37:00.747 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:00 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:00.329+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:37:00.747 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:00 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:00.329+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:37:00.748 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:00 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:00.329+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:37:00.748 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:00 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:00.329+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:27.142218+0000 front 2024-01-24T13:36:27.142234+0000 (oldest deadline 2024-01-24T13:36:51.842262+0000) 2024-01-24T13:37:00.755 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@osd.8.service' 2024-01-24T13:37:00.837 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T13:37:00.837 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-01-24T13:37:00.837 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-01-24T13:37:00.837 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-83afcf5e-babc-11ee-95b1-87774f69a715@osd.9 2024-01-24T13:37:01.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:00 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:00.964+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:37:01.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:00 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:00.964+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:37:01.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:00 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:00.964+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.121624+0000 front 2024-01-24T13:36:12.121460+0000 (oldest deadline 2024-01-24T13:36:37.321997+0000) 2024-01-24T13:37:01.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:00 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:00.964+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:22.623093+0000 front 2024-01-24T13:36:22.623173+0000 (oldest deadline 2024-01-24T13:36:48.523114+0000) 2024-01-24T13:37:01.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:00 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:00.964+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:28.523591+0000 front 2024-01-24T13:36:28.523637+0000 (oldest deadline 2024-01-24T13:36:54.423692+0000) 2024-01-24T13:37:01.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:00 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:00.964+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2024-01-24T13:36:34.424128+0000 front 2024-01-24T13:36:34.424006+0000 (oldest deadline 2024-01-24T13:36:58.524025+0000) 2024-01-24T13:37:01.253 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:00 smithi029 systemd[1]: Stopping Ceph osd.9 for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:37:01.254 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:01 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:37:01.086+0000 7fdcb31df700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T13:37:01.254 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:01 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:37:01.086+0000 7fdcb31df700 -1 osd.9 85 *** Got signal Terminated *** 2024-01-24T13:37:01.254 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:01 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:37:01.086+0000 7fdcb31df700 -1 osd.9 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-24T13:37:01.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:01 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:37:01.254+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:37:01.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:01 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:37:01.254+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.684218+0000 front 2024-01-24T13:36:08.684295+0000 (oldest deadline 2024-01-24T13:36:32.784244+0000) 2024-01-24T13:37:01.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:01 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:37:01.254+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.784926+0000 front 2024-01-24T13:36:12.784800+0000 (oldest deadline 2024-01-24T13:36:38.684866+0000) 2024-01-24T13:37:01.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:01 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:37:01.254+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:18.685211+0000 front 2024-01-24T13:36:18.685291+0000 (oldest deadline 2024-01-24T13:36:48.685987+0000) 2024-01-24T13:37:01.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:01 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:37:01.254+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:29.786971+0000 front 2024-01-24T13:36:29.787029+0000 (oldest deadline 2024-01-24T13:36:51.987063+0000) 2024-01-24T13:37:01.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:01 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:37:01.254+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2024-01-24T13:36:37.188589+0000 front 2024-01-24T13:36:37.188497+0000 (oldest deadline 2024-01-24T13:37:00.088406+0000) 2024-01-24T13:37:01.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:01 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:01.343+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:37:01.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:01 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:01.343+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:37:01.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:01 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:01.343+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:37:01.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:01 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:01.343+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:27.142218+0000 front 2024-01-24T13:36:27.142234+0000 (oldest deadline 2024-01-24T13:36:51.842262+0000) 2024-01-24T13:37:01.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:01 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:01.343+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2024-01-24T13:36:34.743060+0000 front 2024-01-24T13:36:34.743099+0000 (oldest deadline 2024-01-24T13:37:00.643297+0000) 2024-01-24T13:37:02.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:01 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:01.952+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:37:02.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:01 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:01.952+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:37:02.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:01 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:01.952+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.121624+0000 front 2024-01-24T13:36:12.121460+0000 (oldest deadline 2024-01-24T13:36:37.321997+0000) 2024-01-24T13:37:02.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:01 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:01.952+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:22.623093+0000 front 2024-01-24T13:36:22.623173+0000 (oldest deadline 2024-01-24T13:36:48.523114+0000) 2024-01-24T13:37:02.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:01 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:01.952+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:28.523591+0000 front 2024-01-24T13:36:28.523637+0000 (oldest deadline 2024-01-24T13:36:54.423692+0000) 2024-01-24T13:37:02.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:01 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:01.952+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2024-01-24T13:36:34.424128+0000 front 2024-01-24T13:36:34.424006+0000 (oldest deadline 2024-01-24T13:36:58.524025+0000) 2024-01-24T13:37:02.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:02 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:37:02.247+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:37:02.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:02 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:37:02.247+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.684218+0000 front 2024-01-24T13:36:08.684295+0000 (oldest deadline 2024-01-24T13:36:32.784244+0000) 2024-01-24T13:37:02.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:02 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:37:02.247+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.784926+0000 front 2024-01-24T13:36:12.784800+0000 (oldest deadline 2024-01-24T13:36:38.684866+0000) 2024-01-24T13:37:02.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:02 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:37:02.247+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:18.685211+0000 front 2024-01-24T13:36:18.685291+0000 (oldest deadline 2024-01-24T13:36:48.685987+0000) 2024-01-24T13:37:02.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:02 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:37:02.247+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:29.786971+0000 front 2024-01-24T13:36:29.787029+0000 (oldest deadline 2024-01-24T13:36:51.987063+0000) 2024-01-24T13:37:02.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:02 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:37:02.247+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2024-01-24T13:36:37.188589+0000 front 2024-01-24T13:36:37.188497+0000 (oldest deadline 2024-01-24T13:37:00.088406+0000) 2024-01-24T13:37:02.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:02 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:02.349+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:37:02.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:02 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:02.349+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:37:02.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:02 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:02.349+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:37:02.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:02 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:02.349+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:27.142218+0000 front 2024-01-24T13:36:27.142234+0000 (oldest deadline 2024-01-24T13:36:51.842262+0000) 2024-01-24T13:37:02.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:02 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:02.349+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2024-01-24T13:36:34.743060+0000 front 2024-01-24T13:36:34.743099+0000 (oldest deadline 2024-01-24T13:37:00.643297+0000) 2024-01-24T13:37:03.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:02 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:02.920+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:37:03.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:02 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:02.920+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:37:03.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:02 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:02.920+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.121624+0000 front 2024-01-24T13:36:12.121460+0000 (oldest deadline 2024-01-24T13:36:37.321997+0000) 2024-01-24T13:37:03.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:02 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:02.920+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:22.623093+0000 front 2024-01-24T13:36:22.623173+0000 (oldest deadline 2024-01-24T13:36:48.523114+0000) 2024-01-24T13:37:03.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:02 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:02.920+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:28.523591+0000 front 2024-01-24T13:36:28.523637+0000 (oldest deadline 2024-01-24T13:36:54.423692+0000) 2024-01-24T13:37:03.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:02 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:02.920+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2024-01-24T13:36:34.424128+0000 front 2024-01-24T13:36:34.424006+0000 (oldest deadline 2024-01-24T13:36:58.524025+0000) 2024-01-24T13:37:03.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:03 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:37:03.200+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:37:03.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:03 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:37:03.200+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.684218+0000 front 2024-01-24T13:36:08.684295+0000 (oldest deadline 2024-01-24T13:36:32.784244+0000) 2024-01-24T13:37:03.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:03 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:37:03.200+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.784926+0000 front 2024-01-24T13:36:12.784800+0000 (oldest deadline 2024-01-24T13:36:38.684866+0000) 2024-01-24T13:37:03.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:03 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:37:03.200+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:18.685211+0000 front 2024-01-24T13:36:18.685291+0000 (oldest deadline 2024-01-24T13:36:48.685987+0000) 2024-01-24T13:37:03.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:03 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:37:03.200+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:29.786971+0000 front 2024-01-24T13:36:29.787029+0000 (oldest deadline 2024-01-24T13:36:51.987063+0000) 2024-01-24T13:37:03.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:03 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:37:03.200+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2024-01-24T13:36:37.188589+0000 front 2024-01-24T13:36:37.188497+0000 (oldest deadline 2024-01-24T13:37:00.088406+0000) 2024-01-24T13:37:03.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:03 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:03.320+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:37:03.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:03 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:03.320+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:37:03.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:03 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:03.320+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:37:03.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:03 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:03.320+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:27.142218+0000 front 2024-01-24T13:36:27.142234+0000 (oldest deadline 2024-01-24T13:36:51.842262+0000) 2024-01-24T13:37:03.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:03 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:03.320+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2024-01-24T13:36:34.743060+0000 front 2024-01-24T13:36:34.743099+0000 (oldest deadline 2024-01-24T13:37:00.643297+0000) 2024-01-24T13:37:04.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:03 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:03.884+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:37:04.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:03 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:03.884+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:37:04.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:03 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:03.884+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.121624+0000 front 2024-01-24T13:36:12.121460+0000 (oldest deadline 2024-01-24T13:36:37.321997+0000) 2024-01-24T13:37:04.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:03 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:03.884+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:22.623093+0000 front 2024-01-24T13:36:22.623173+0000 (oldest deadline 2024-01-24T13:36:48.523114+0000) 2024-01-24T13:37:04.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:03 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:03.884+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:28.523591+0000 front 2024-01-24T13:36:28.523637+0000 (oldest deadline 2024-01-24T13:36:54.423692+0000) 2024-01-24T13:37:04.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:03 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:03.884+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2024-01-24T13:36:34.424128+0000 front 2024-01-24T13:36:34.424006+0000 (oldest deadline 2024-01-24T13:36:58.524025+0000) 2024-01-24T13:37:04.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:04 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:37:04.192+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:37:04.677 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:04 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:37:04.192+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.684218+0000 front 2024-01-24T13:36:08.684295+0000 (oldest deadline 2024-01-24T13:36:32.784244+0000) 2024-01-24T13:37:04.677 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:04 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:37:04.192+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.784926+0000 front 2024-01-24T13:36:12.784800+0000 (oldest deadline 2024-01-24T13:36:38.684866+0000) 2024-01-24T13:37:04.677 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:04 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:37:04.192+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:18.685211+0000 front 2024-01-24T13:36:18.685291+0000 (oldest deadline 2024-01-24T13:36:48.685987+0000) 2024-01-24T13:37:04.678 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:04 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:37:04.192+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:29.786971+0000 front 2024-01-24T13:36:29.787029+0000 (oldest deadline 2024-01-24T13:36:51.987063+0000) 2024-01-24T13:37:04.678 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:04 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:37:04.192+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2024-01-24T13:36:37.188589+0000 front 2024-01-24T13:36:37.188497+0000 (oldest deadline 2024-01-24T13:37:00.088406+0000) 2024-01-24T13:37:04.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:04 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:04.337+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:37:04.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:04 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:04.337+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:37:04.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:04 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:04.337+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:37:04.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:04 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:04.337+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:27.142218+0000 front 2024-01-24T13:36:27.142234+0000 (oldest deadline 2024-01-24T13:36:51.842262+0000) 2024-01-24T13:37:04.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:04 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:04.337+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2024-01-24T13:36:34.743060+0000 front 2024-01-24T13:36:34.743099+0000 (oldest deadline 2024-01-24T13:37:00.643297+0000) 2024-01-24T13:37:05.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:04 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:04.873+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:37:05.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:04 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:04.873+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:37:05.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:04 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:04.873+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.121624+0000 front 2024-01-24T13:36:12.121460+0000 (oldest deadline 2024-01-24T13:36:37.321997+0000) 2024-01-24T13:37:05.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:04 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:04.873+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:22.623093+0000 front 2024-01-24T13:36:22.623173+0000 (oldest deadline 2024-01-24T13:36:48.523114+0000) 2024-01-24T13:37:05.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:04 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:04.873+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:28.523591+0000 front 2024-01-24T13:36:28.523637+0000 (oldest deadline 2024-01-24T13:36:54.423692+0000) 2024-01-24T13:37:05.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:04 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:04.873+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2024-01-24T13:36:34.424128+0000 front 2024-01-24T13:36:34.424006+0000 (oldest deadline 2024-01-24T13:36:58.524025+0000) 2024-01-24T13:37:05.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:05 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:37:05.171+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:00.681999+0000 front 2024-01-24T13:36:00.682081+0000 (oldest deadline 2024-01-24T13:36:23.582135+0000) 2024-01-24T13:37:05.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:05 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:37:05.171+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.684218+0000 front 2024-01-24T13:36:08.684295+0000 (oldest deadline 2024-01-24T13:36:32.784244+0000) 2024-01-24T13:37:05.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:05 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:37:05.171+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.784926+0000 front 2024-01-24T13:36:12.784800+0000 (oldest deadline 2024-01-24T13:36:38.684866+0000) 2024-01-24T13:37:05.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:05 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:37:05.171+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:18.685211+0000 front 2024-01-24T13:36:18.685291+0000 (oldest deadline 2024-01-24T13:36:48.685987+0000) 2024-01-24T13:37:05.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:05 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:37:05.171+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:29.786971+0000 front 2024-01-24T13:36:29.787029+0000 (oldest deadline 2024-01-24T13:36:51.987063+0000) 2024-01-24T13:37:05.599 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:05 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9[84571]: 2024-01-24T13:37:05.171+0000 7fdcacfd0700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2024-01-24T13:36:37.188589+0000 front 2024-01-24T13:36:37.188497+0000 (oldest deadline 2024-01-24T13:37:00.088406+0000) 2024-01-24T13:37:05.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:05 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:05.332+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:37:05.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:05 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:05.332+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:37:05.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:05 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:05.332+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:37:05.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:05 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:05.332+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:27.142218+0000 front 2024-01-24T13:36:27.142234+0000 (oldest deadline 2024-01-24T13:36:51.842262+0000) 2024-01-24T13:37:05.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:05 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:05.332+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2024-01-24T13:36:34.743060+0000 front 2024-01-24T13:36:34.743099+0000 (oldest deadline 2024-01-24T13:37:00.643297+0000) 2024-01-24T13:37:06.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:05 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:05.887+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:37:06.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:05 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:05.887+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:37:06.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:05 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:05.887+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.121624+0000 front 2024-01-24T13:36:12.121460+0000 (oldest deadline 2024-01-24T13:36:37.321997+0000) 2024-01-24T13:37:06.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:05 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:05.887+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:22.623093+0000 front 2024-01-24T13:36:22.623173+0000 (oldest deadline 2024-01-24T13:36:48.523114+0000) 2024-01-24T13:37:06.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:05 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:05.887+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:28.523591+0000 front 2024-01-24T13:36:28.523637+0000 (oldest deadline 2024-01-24T13:36:54.423692+0000) 2024-01-24T13:37:06.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:05 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:05.887+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2024-01-24T13:36:34.424128+0000 front 2024-01-24T13:36:34.424006+0000 (oldest deadline 2024-01-24T13:36:58.524025+0000) 2024-01-24T13:37:06.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:06 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:06.356+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:37:06.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:06 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:06.356+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:37:06.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:06 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:06.356+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:37:06.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:06 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:06.356+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:27.142218+0000 front 2024-01-24T13:36:27.142234+0000 (oldest deadline 2024-01-24T13:36:51.842262+0000) 2024-01-24T13:37:06.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:06 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:06.356+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2024-01-24T13:36:34.743060+0000 front 2024-01-24T13:36:34.743099+0000 (oldest deadline 2024-01-24T13:37:00.643297+0000) 2024-01-24T13:37:06.848 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jan 24 13:37:06 smithi029 bash[97492]: ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-9 2024-01-24T13:37:07.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:06 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:06.870+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:37:07.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:06 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:06.870+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:37:07.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:06 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:06.870+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.121624+0000 front 2024-01-24T13:36:12.121460+0000 (oldest deadline 2024-01-24T13:36:37.321997+0000) 2024-01-24T13:37:07.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:06 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:06.870+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:22.623093+0000 front 2024-01-24T13:36:22.623173+0000 (oldest deadline 2024-01-24T13:36:48.523114+0000) 2024-01-24T13:37:07.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:06 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:06.870+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:28.523591+0000 front 2024-01-24T13:36:28.523637+0000 (oldest deadline 2024-01-24T13:36:54.423692+0000) 2024-01-24T13:37:07.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:06 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:06.870+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2024-01-24T13:36:34.424128+0000 front 2024-01-24T13:36:34.424006+0000 (oldest deadline 2024-01-24T13:36:58.524025+0000) 2024-01-24T13:37:07.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:07 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:07.386+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:37:07.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:07 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:07.386+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:37:07.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:07 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:07.386+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:37:07.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:07 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:07.386+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:27.142218+0000 front 2024-01-24T13:36:27.142234+0000 (oldest deadline 2024-01-24T13:36:51.842262+0000) 2024-01-24T13:37:07.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:07 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:07.386+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2024-01-24T13:36:34.743060+0000 front 2024-01-24T13:36:34.743099+0000 (oldest deadline 2024-01-24T13:37:00.643297+0000) 2024-01-24T13:37:07.867 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@osd.9.service' 2024-01-24T13:37:07.918 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T13:37:07.919 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-01-24T13:37:07.919 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-01-24T13:37:07.919 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-83afcf5e-babc-11ee-95b1-87774f69a715@osd.10 2024-01-24T13:37:08.163 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:07 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:07.900+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:37:08.164 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:07 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:07.900+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:37:08.164 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:07 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:07.900+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.121624+0000 front 2024-01-24T13:36:12.121460+0000 (oldest deadline 2024-01-24T13:36:37.321997+0000) 2024-01-24T13:37:08.164 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:07 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:07.900+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:22.623093+0000 front 2024-01-24T13:36:22.623173+0000 (oldest deadline 2024-01-24T13:36:48.523114+0000) 2024-01-24T13:37:08.164 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:07 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:07.900+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:28.523591+0000 front 2024-01-24T13:36:28.523637+0000 (oldest deadline 2024-01-24T13:36:54.423692+0000) 2024-01-24T13:37:08.164 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:07 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:07.900+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2024-01-24T13:36:34.424128+0000 front 2024-01-24T13:36:34.424006+0000 (oldest deadline 2024-01-24T13:36:58.524025+0000) 2024-01-24T13:37:08.164 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:07 smithi163 systemd[1]: Stopping Ceph osd.10 for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:37:08.164 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:08 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:08.162+0000 7f2645d97700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T13:37:08.481 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:08 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:08.162+0000 7f2645d97700 -1 osd.10 85 *** Got signal Terminated *** 2024-01-24T13:37:08.482 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:08 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:08.162+0000 7f2645d97700 -1 osd.10 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-24T13:37:08.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:08 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:08.431+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:37:08.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:08 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:08.431+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:37:08.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:08 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:08.431+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:37:08.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:08 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:08.431+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:27.142218+0000 front 2024-01-24T13:36:27.142234+0000 (oldest deadline 2024-01-24T13:36:51.842262+0000) 2024-01-24T13:37:08.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:08 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:08.431+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2024-01-24T13:36:34.743060+0000 front 2024-01-24T13:36:34.743099+0000 (oldest deadline 2024-01-24T13:37:00.643297+0000) 2024-01-24T13:37:09.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:08 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:08.883+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:37:09.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:08 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:08.883+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:37:09.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:08 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:08.883+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.121624+0000 front 2024-01-24T13:36:12.121460+0000 (oldest deadline 2024-01-24T13:36:37.321997+0000) 2024-01-24T13:37:09.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:08 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:08.883+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:22.623093+0000 front 2024-01-24T13:36:22.623173+0000 (oldest deadline 2024-01-24T13:36:48.523114+0000) 2024-01-24T13:37:09.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:08 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:08.883+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:28.523591+0000 front 2024-01-24T13:36:28.523637+0000 (oldest deadline 2024-01-24T13:36:54.423692+0000) 2024-01-24T13:37:09.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:08 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:08.883+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2024-01-24T13:36:34.424128+0000 front 2024-01-24T13:36:34.424006+0000 (oldest deadline 2024-01-24T13:36:58.524025+0000) 2024-01-24T13:37:09.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:09 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:09.423+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:37:09.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:09 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:09.423+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:37:09.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:09 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:09.423+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:37:09.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:09 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:09.423+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:27.142218+0000 front 2024-01-24T13:36:27.142234+0000 (oldest deadline 2024-01-24T13:36:51.842262+0000) 2024-01-24T13:37:09.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:09 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:09.423+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2024-01-24T13:36:34.743060+0000 front 2024-01-24T13:36:34.743099+0000 (oldest deadline 2024-01-24T13:37:00.643297+0000) 2024-01-24T13:37:09.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:09 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:09.423+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-01-24T13:36:44.044849+0000 front 2024-01-24T13:36:44.044730+0000 (oldest deadline 2024-01-24T13:37:09.344618+0000) 2024-01-24T13:37:10.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:09 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:09.900+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:37:10.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:09 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:09.900+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:37:10.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:09 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:09.900+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.121624+0000 front 2024-01-24T13:36:12.121460+0000 (oldest deadline 2024-01-24T13:36:37.321997+0000) 2024-01-24T13:37:10.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:09 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:09.900+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:22.623093+0000 front 2024-01-24T13:36:22.623173+0000 (oldest deadline 2024-01-24T13:36:48.523114+0000) 2024-01-24T13:37:10.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:09 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:09.900+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:28.523591+0000 front 2024-01-24T13:36:28.523637+0000 (oldest deadline 2024-01-24T13:36:54.423692+0000) 2024-01-24T13:37:10.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:09 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:09.900+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2024-01-24T13:36:34.424128+0000 front 2024-01-24T13:36:34.424006+0000 (oldest deadline 2024-01-24T13:36:58.524025+0000) 2024-01-24T13:37:10.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:09 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:09.900+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-01-24T13:36:44.325619+0000 front 2024-01-24T13:36:44.325743+0000 (oldest deadline 2024-01-24T13:37:09.025507+0000) 2024-01-24T13:37:10.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:10 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:10.420+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:37:10.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:10 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:10.420+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:37:10.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:10 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:10.420+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:37:10.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:10 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:10.420+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:27.142218+0000 front 2024-01-24T13:36:27.142234+0000 (oldest deadline 2024-01-24T13:36:51.842262+0000) 2024-01-24T13:37:10.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:10 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:10.420+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2024-01-24T13:36:34.743060+0000 front 2024-01-24T13:36:34.743099+0000 (oldest deadline 2024-01-24T13:37:00.643297+0000) 2024-01-24T13:37:10.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:10 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:10.420+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-01-24T13:36:44.044849+0000 front 2024-01-24T13:36:44.044730+0000 (oldest deadline 2024-01-24T13:37:09.344618+0000) 2024-01-24T13:37:11.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:10 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:10.940+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:37:11.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:10 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:10.940+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:37:11.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:10 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:10.940+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.121624+0000 front 2024-01-24T13:36:12.121460+0000 (oldest deadline 2024-01-24T13:36:37.321997+0000) 2024-01-24T13:37:11.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:10 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:10.940+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:22.623093+0000 front 2024-01-24T13:36:22.623173+0000 (oldest deadline 2024-01-24T13:36:48.523114+0000) 2024-01-24T13:37:11.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:10 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:10.940+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:28.523591+0000 front 2024-01-24T13:36:28.523637+0000 (oldest deadline 2024-01-24T13:36:54.423692+0000) 2024-01-24T13:37:11.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:10 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:10.940+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2024-01-24T13:36:34.424128+0000 front 2024-01-24T13:36:34.424006+0000 (oldest deadline 2024-01-24T13:36:58.524025+0000) 2024-01-24T13:37:11.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:10 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:10.940+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-01-24T13:36:44.325619+0000 front 2024-01-24T13:36:44.325743+0000 (oldest deadline 2024-01-24T13:37:09.025507+0000) 2024-01-24T13:37:11.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:11 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:11.393+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:37:11.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:11 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:11.393+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:37:11.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:11 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:11.393+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:37:11.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:11 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:11.393+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:27.142218+0000 front 2024-01-24T13:36:27.142234+0000 (oldest deadline 2024-01-24T13:36:51.842262+0000) 2024-01-24T13:37:11.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:11 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:11.393+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2024-01-24T13:36:34.743060+0000 front 2024-01-24T13:36:34.743099+0000 (oldest deadline 2024-01-24T13:37:00.643297+0000) 2024-01-24T13:37:11.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:11 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:11.393+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-01-24T13:36:44.044849+0000 front 2024-01-24T13:36:44.044730+0000 (oldest deadline 2024-01-24T13:37:09.344618+0000) 2024-01-24T13:37:12.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:11 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:11.972+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:37:12.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:11 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:11.972+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:37:12.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:11 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:11.972+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.121624+0000 front 2024-01-24T13:36:12.121460+0000 (oldest deadline 2024-01-24T13:36:37.321997+0000) 2024-01-24T13:37:12.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:11 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:11.972+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:22.623093+0000 front 2024-01-24T13:36:22.623173+0000 (oldest deadline 2024-01-24T13:36:48.523114+0000) 2024-01-24T13:37:12.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:11 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:11.972+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:28.523591+0000 front 2024-01-24T13:36:28.523637+0000 (oldest deadline 2024-01-24T13:36:54.423692+0000) 2024-01-24T13:37:12.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:11 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:11.972+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2024-01-24T13:36:34.424128+0000 front 2024-01-24T13:36:34.424006+0000 (oldest deadline 2024-01-24T13:36:58.524025+0000) 2024-01-24T13:37:12.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:11 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:11.972+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-01-24T13:36:44.325619+0000 front 2024-01-24T13:36:44.325743+0000 (oldest deadline 2024-01-24T13:37:09.025507+0000) 2024-01-24T13:37:12.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:12 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:12.350+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:37:12.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:12 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:12.350+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:37:12.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:12 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:12.350+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:37:12.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:12 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:12.350+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:27.142218+0000 front 2024-01-24T13:36:27.142234+0000 (oldest deadline 2024-01-24T13:36:51.842262+0000) 2024-01-24T13:37:12.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:12 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:12.350+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2024-01-24T13:36:34.743060+0000 front 2024-01-24T13:36:34.743099+0000 (oldest deadline 2024-01-24T13:37:00.643297+0000) 2024-01-24T13:37:12.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:12 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:12.350+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-01-24T13:36:44.044849+0000 front 2024-01-24T13:36:44.044730+0000 (oldest deadline 2024-01-24T13:37:09.344618+0000) 2024-01-24T13:37:13.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:12 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:12.928+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:35:59.819812+0000 front 2024-01-24T13:35:59.819729+0000 (oldest deadline 2024-01-24T13:36:24.519769+0000) 2024-01-24T13:37:13.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:12 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:12.928+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:07.420711+0000 front 2024-01-24T13:36:07.420734+0000 (oldest deadline 2024-01-24T13:36:32.120911+0000) 2024-01-24T13:37:13.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:12 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:12.928+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:12.121624+0000 front 2024-01-24T13:36:12.121460+0000 (oldest deadline 2024-01-24T13:36:37.321997+0000) 2024-01-24T13:37:13.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:12 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:12.928+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-01-24T13:36:22.623093+0000 front 2024-01-24T13:36:22.623173+0000 (oldest deadline 2024-01-24T13:36:48.523114+0000) 2024-01-24T13:37:13.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:12 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:12.928+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:28.523591+0000 front 2024-01-24T13:36:28.523637+0000 (oldest deadline 2024-01-24T13:36:54.423692+0000) 2024-01-24T13:37:13.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:12 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:12.928+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2024-01-24T13:36:34.424128+0000 front 2024-01-24T13:36:34.424006+0000 (oldest deadline 2024-01-24T13:36:58.524025+0000) 2024-01-24T13:37:13.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:12 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10[75877]: 2024-01-24T13:37:12.928+0000 7f263fb88700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-01-24T13:36:44.325619+0000 front 2024-01-24T13:36:44.325743+0000 (oldest deadline 2024-01-24T13:37:09.025507+0000) 2024-01-24T13:37:13.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:13 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:13.339+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:37:13.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:13 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:13.339+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:37:13.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:13 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:13.339+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:37:13.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:13 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:13.339+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:27.142218+0000 front 2024-01-24T13:36:27.142234+0000 (oldest deadline 2024-01-24T13:36:51.842262+0000) 2024-01-24T13:37:13.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:13 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:13.339+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2024-01-24T13:36:34.743060+0000 front 2024-01-24T13:36:34.743099+0000 (oldest deadline 2024-01-24T13:37:00.643297+0000) 2024-01-24T13:37:13.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:13 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:13.339+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-01-24T13:36:44.044849+0000 front 2024-01-24T13:36:44.044730+0000 (oldest deadline 2024-01-24T13:37:09.344618+0000) 2024-01-24T13:37:14.232 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jan 24 13:37:13 smithi163 bash[85348]: ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-10 2024-01-24T13:37:14.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:14 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:14.347+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:37:14.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:14 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:14.347+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:37:14.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:14 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:14.347+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:37:14.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:14 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:14.347+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:27.142218+0000 front 2024-01-24T13:36:27.142234+0000 (oldest deadline 2024-01-24T13:36:51.842262+0000) 2024-01-24T13:37:14.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:14 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:14.347+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2024-01-24T13:36:34.743060+0000 front 2024-01-24T13:36:34.743099+0000 (oldest deadline 2024-01-24T13:37:00.643297+0000) 2024-01-24T13:37:14.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:14 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:14.347+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-01-24T13:36:44.044849+0000 front 2024-01-24T13:36:44.044730+0000 (oldest deadline 2024-01-24T13:37:09.344618+0000) 2024-01-24T13:37:14.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:14 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:14.347+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6820 osd.7 since back 2024-01-24T13:36:50.445637+0000 front 2024-01-24T13:36:50.445654+0000 (oldest deadline 2024-01-24T13:37:13.345470+0000) 2024-01-24T13:37:15.042 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@osd.10.service' 2024-01-24T13:37:15.084 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T13:37:15.084 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-01-24T13:37:15.084 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-01-24T13:37:15.084 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-83afcf5e-babc-11ee-95b1-87774f69a715@osd.11 2024-01-24T13:37:15.555 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:15 smithi187 systemd[1]: Stopping Ceph osd.11 for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:37:15.555 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:15 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:15.308+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:37:15.555 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:15 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:15.308+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:37:15.555 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:15 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:15.308+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:37:15.555 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:15 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:15.308+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:27.142218+0000 front 2024-01-24T13:36:27.142234+0000 (oldest deadline 2024-01-24T13:36:51.842262+0000) 2024-01-24T13:37:15.555 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:15 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:15.308+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2024-01-24T13:36:34.743060+0000 front 2024-01-24T13:36:34.743099+0000 (oldest deadline 2024-01-24T13:37:00.643297+0000) 2024-01-24T13:37:15.555 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:15 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:15.308+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-01-24T13:36:44.044849+0000 front 2024-01-24T13:36:44.044730+0000 (oldest deadline 2024-01-24T13:37:09.344618+0000) 2024-01-24T13:37:15.555 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:15 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:15.308+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6820 osd.7 since back 2024-01-24T13:36:50.445637+0000 front 2024-01-24T13:36:50.445654+0000 (oldest deadline 2024-01-24T13:37:13.345470+0000) 2024-01-24T13:37:15.555 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:15 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:15.330+0000 7fe846e42700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T13:37:15.555 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:15 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:15.330+0000 7fe846e42700 -1 osd.11 85 *** Got signal Terminated *** 2024-01-24T13:37:15.555 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:15 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:15.330+0000 7fe846e42700 -1 osd.11 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-24T13:37:16.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:16 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:16.306+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:37:16.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:16 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:16.306+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:37:16.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:16 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:16.306+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:37:16.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:16 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:16.306+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:27.142218+0000 front 2024-01-24T13:36:27.142234+0000 (oldest deadline 2024-01-24T13:36:51.842262+0000) 2024-01-24T13:37:16.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:16 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:16.306+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2024-01-24T13:36:34.743060+0000 front 2024-01-24T13:36:34.743099+0000 (oldest deadline 2024-01-24T13:37:00.643297+0000) 2024-01-24T13:37:16.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:16 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:16.306+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-01-24T13:36:44.044849+0000 front 2024-01-24T13:36:44.044730+0000 (oldest deadline 2024-01-24T13:37:09.344618+0000) 2024-01-24T13:37:16.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:16 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:16.306+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6820 osd.7 since back 2024-01-24T13:36:50.445637+0000 front 2024-01-24T13:36:50.445654+0000 (oldest deadline 2024-01-24T13:37:13.345470+0000) 2024-01-24T13:37:17.554 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:17 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:17.290+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:37:17.555 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:17 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:17.290+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:37:17.555 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:17 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:17.290+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:37:17.555 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:17 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:17.290+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:27.142218+0000 front 2024-01-24T13:36:27.142234+0000 (oldest deadline 2024-01-24T13:36:51.842262+0000) 2024-01-24T13:37:17.555 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:17 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:17.290+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2024-01-24T13:36:34.743060+0000 front 2024-01-24T13:36:34.743099+0000 (oldest deadline 2024-01-24T13:37:00.643297+0000) 2024-01-24T13:37:17.555 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:17 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:17.290+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-01-24T13:36:44.044849+0000 front 2024-01-24T13:36:44.044730+0000 (oldest deadline 2024-01-24T13:37:09.344618+0000) 2024-01-24T13:37:17.555 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:17 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:17.290+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6820 osd.7 since back 2024-01-24T13:36:50.445637+0000 front 2024-01-24T13:36:50.445654+0000 (oldest deadline 2024-01-24T13:37:13.345470+0000) 2024-01-24T13:37:18.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:18 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:18.330+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:37:18.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:18 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:18.330+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:37:18.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:18 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:18.330+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:37:18.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:18 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:18.330+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:27.142218+0000 front 2024-01-24T13:36:27.142234+0000 (oldest deadline 2024-01-24T13:36:51.842262+0000) 2024-01-24T13:37:18.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:18 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:18.330+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2024-01-24T13:36:34.743060+0000 front 2024-01-24T13:36:34.743099+0000 (oldest deadline 2024-01-24T13:37:00.643297+0000) 2024-01-24T13:37:18.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:18 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:18.330+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-01-24T13:36:44.044849+0000 front 2024-01-24T13:36:44.044730+0000 (oldest deadline 2024-01-24T13:37:09.344618+0000) 2024-01-24T13:37:18.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:18 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:18.330+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6820 osd.7 since back 2024-01-24T13:36:50.445637+0000 front 2024-01-24T13:36:50.445654+0000 (oldest deadline 2024-01-24T13:37:13.345470+0000) 2024-01-24T13:37:19.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:19 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:19.328+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-01-24T13:36:01.137284+0000 front 2024-01-24T13:36:01.137329+0000 (oldest deadline 2024-01-24T13:36:23.437311+0000) 2024-01-24T13:37:19.804 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:19 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:19.328+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6804 osd.1 since back 2024-01-24T13:36:08.039063+0000 front 2024-01-24T13:36:08.038950+0000 (oldest deadline 2024-01-24T13:36:32.738987+0000) 2024-01-24T13:37:19.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:19 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:19.328+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2024-01-24T13:36:14.940499+0000 front 2024-01-24T13:36:14.940381+0000 (oldest deadline 2024-01-24T13:36:37.240494+0000) 2024-01-24T13:37:19.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:19 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:19.328+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-01-24T13:36:27.142218+0000 front 2024-01-24T13:36:27.142234+0000 (oldest deadline 2024-01-24T13:36:51.842262+0000) 2024-01-24T13:37:19.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:19 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:19.328+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2024-01-24T13:36:34.743060+0000 front 2024-01-24T13:36:34.743099+0000 (oldest deadline 2024-01-24T13:37:00.643297+0000) 2024-01-24T13:37:19.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:19 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:19.328+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-01-24T13:36:44.044849+0000 front 2024-01-24T13:36:44.044730+0000 (oldest deadline 2024-01-24T13:37:09.344618+0000) 2024-01-24T13:37:19.805 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:19 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11[75324]: 2024-01-24T13:37:19.328+0000 7fe841c35700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.163:6820 osd.7 since back 2024-01-24T13:36:50.445637+0000 front 2024-01-24T13:36:50.445654+0000 (oldest deadline 2024-01-24T13:37:13.345470+0000) 2024-01-24T13:37:21.304 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jan 24 13:37:20 smithi187 bash[84182]: ceph-83afcf5e-babc-11ee-95b1-87774f69a715-osd-11 2024-01-24T13:37:22.092 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@osd.11.service' 2024-01-24T13:37:22.133 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T13:37:22.133 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-01-24T13:37:22.133 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-01-24T13:37:22.133 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mds.a 2024-01-24T13:37:22.599 INFO:journalctl@ceph.mds.a.smithi029.stdout:Jan 24 13:37:22 smithi029 systemd[1]: Stopping Ceph mds.a for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:37:22.599 INFO:journalctl@ceph.mds.a.smithi029.stdout:Jan 24 13:37:22 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-a[92459]: 2024-01-24T13:37:22.392+0000 7f3f0adda700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T13:37:22.599 INFO:journalctl@ceph.mds.a.smithi029.stdout:Jan 24 13:37:22 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-a[92459]: 2024-01-24T13:37:22.392+0000 7f3f0adda700 -1 mds.a *** got signal Terminated *** 2024-01-24T13:37:26.849 INFO:journalctl@ceph.mds.a.smithi029.stdout:Jan 24 13:37:26 smithi029 bash[97747]: ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-a 2024-01-24T13:37:26.950 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mds.a.service' 2024-01-24T13:37:26.991 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T13:37:26.991 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-01-24T13:37:26.992 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-01-24T13:37:26.992 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mds.d 2024-01-24T13:37:27.274 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jan 24 13:37:27 smithi029 systemd[1]: Stopping Ceph mds.d for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:37:27.599 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jan 24 13:37:27 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-d[91550]: 2024-01-24T13:37:27.273+0000 7fdcb92c2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T13:37:27.599 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jan 24 13:37:27 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-d[91550]: 2024-01-24T13:37:27.273+0000 7fdcb92c2700 -1 mds.d *** got signal Terminated *** 2024-01-24T13:37:30.599 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jan 24 13:37:30 smithi029 bash[97887]: ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-d 2024-01-24T13:37:30.709 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mds.d.service' 2024-01-24T13:37:30.747 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T13:37:30.748 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-01-24T13:37:30.748 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-01-24T13:37:30.748 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mds.g 2024-01-24T13:37:31.083 INFO:journalctl@ceph.mds.g.smithi029.stdout:Jan 24 13:37:30 smithi029 systemd[1]: Stopping Ceph mds.g for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:37:31.349 INFO:journalctl@ceph.mds.g.smithi029.stdout:Jan 24 13:37:31 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-g[92070]: 2024-01-24T13:37:31.081+0000 7fad6fd62700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T13:37:31.349 INFO:journalctl@ceph.mds.g.smithi029.stdout:Jan 24 13:37:31 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-g[92070]: 2024-01-24T13:37:31.081+0000 7fad6fd62700 -1 mds.g *** got signal Terminated *** 2024-01-24T13:37:33.349 INFO:journalctl@ceph.mds.g.smithi029.stdout:Jan 24 13:37:33 smithi029 bash[98026]: ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-g 2024-01-24T13:37:33.392 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mds.g.service' 2024-01-24T13:37:33.430 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T13:37:33.430 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-01-24T13:37:33.430 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-01-24T13:37:33.430 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mds.j 2024-01-24T13:37:33.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jan 24 13:37:33 smithi029 systemd[1]: Stopping Ceph mds.j for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:37:34.099 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jan 24 13:37:33 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-j[91163]: 2024-01-24T13:37:33.714+0000 7f40c989c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T13:37:34.099 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jan 24 13:37:33 smithi029 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-j[91163]: 2024-01-24T13:37:33.714+0000 7f40c989c700 -1 mds.j *** got signal Terminated *** 2024-01-24T13:37:36.849 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jan 24 13:37:36 smithi029 bash[98166]: ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-j 2024-01-24T13:37:36.999 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mds.j.service' 2024-01-24T13:37:37.037 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T13:37:37.037 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-01-24T13:37:37.038 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-01-24T13:37:37.038 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mds.b 2024-01-24T13:37:37.482 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jan 24 13:37:37 smithi163 systemd[1]: Stopping Ceph mds.b for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:37:37.482 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jan 24 13:37:37 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-b[82918]: 2024-01-24T13:37:37.282+0000 7ff9d4b1b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T13:37:37.482 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jan 24 13:37:37 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-b[82918]: 2024-01-24T13:37:37.282+0000 7ff9d4b1b700 -1 mds.b *** got signal Terminated *** 2024-01-24T13:37:41.482 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jan 24 13:37:41 smithi163 bash[85612]: ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-b 2024-01-24T13:37:41.511 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mds.b.service' 2024-01-24T13:37:41.547 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T13:37:41.547 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-01-24T13:37:41.547 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-01-24T13:37:41.547 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mds.e 2024-01-24T13:37:41.829 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jan 24 13:37:41 smithi163 systemd[1]: Stopping Ceph mds.e for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:37:42.232 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jan 24 13:37:41 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-e[82237]: 2024-01-24T13:37:41.828+0000 7f0b84660700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T13:37:42.232 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jan 24 13:37:41 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-e[82237]: 2024-01-24T13:37:41.828+0000 7f0b84660700 -1 mds.e *** got signal Terminated *** 2024-01-24T13:37:44.402 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jan 24 13:37:44 smithi163 bash[85750]: ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-e 2024-01-24T13:37:44.407 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mds.e.service' 2024-01-24T13:37:44.441 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T13:37:44.442 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-01-24T13:37:44.442 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-01-24T13:37:44.442 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mds.h 2024-01-24T13:37:44.714 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jan 24 13:37:44 smithi163 systemd[1]: Stopping Ceph mds.h for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:37:44.982 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jan 24 13:37:44 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-h[82574]: 2024-01-24T13:37:44.712+0000 7f7eccfd8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T13:37:44.982 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jan 24 13:37:44 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-h[82574]: 2024-01-24T13:37:44.712+0000 7f7eccfd8700 -1 mds.h *** got signal Terminated *** 2024-01-24T13:37:47.435 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jan 24 13:37:47 smithi163 bash[85890]: ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-h 2024-01-24T13:37:47.440 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mds.h.service' 2024-01-24T13:37:47.474 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T13:37:47.474 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-01-24T13:37:47.475 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-01-24T13:37:47.475 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mds.k 2024-01-24T13:37:47.732 INFO:journalctl@ceph.mds.k.smithi163.stdout:Jan 24 13:37:47 smithi163 systemd[1]: Stopping Ceph mds.k for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:37:48.232 INFO:journalctl@ceph.mds.k.smithi163.stdout:Jan 24 13:37:47 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-k[81886]: 2024-01-24T13:37:47.738+0000 7f4d65093700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T13:37:48.232 INFO:journalctl@ceph.mds.k.smithi163.stdout:Jan 24 13:37:47 smithi163 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-k[81886]: 2024-01-24T13:37:47.738+0000 7f4d65093700 -1 mds.k *** got signal Terminated *** 2024-01-24T13:37:50.982 INFO:journalctl@ceph.mds.k.smithi163.stdout:Jan 24 13:37:50 smithi163 bash[86030]: ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-k 2024-01-24T13:37:51.106 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mds.k.service' 2024-01-24T13:37:51.139 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T13:37:51.139 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-01-24T13:37:51.139 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-01-24T13:37:51.140 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mds.c 2024-01-24T13:37:51.555 INFO:journalctl@ceph.mds.c.smithi187.stdout:Jan 24 13:37:51 smithi187 systemd[1]: Stopping Ceph mds.c for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:37:51.555 INFO:journalctl@ceph.mds.c.smithi187.stdout:Jan 24 13:37:51 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-c[81381]: 2024-01-24T13:37:51.374+0000 7f3ea7fb0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T13:37:51.555 INFO:journalctl@ceph.mds.c.smithi187.stdout:Jan 24 13:37:51 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-c[81381]: 2024-01-24T13:37:51.374+0000 7f3ea7fb0700 -1 mds.c *** got signal Terminated *** 2024-01-24T13:37:55.804 INFO:journalctl@ceph.mds.c.smithi187.stdout:Jan 24 13:37:55 smithi187 bash[84439]: ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-c 2024-01-24T13:37:55.870 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mds.c.service' 2024-01-24T13:37:55.906 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T13:37:55.906 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-01-24T13:37:55.906 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-01-24T13:37:55.907 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mds.f 2024-01-24T13:37:56.173 INFO:journalctl@ceph.mds.f.smithi187.stdout:Jan 24 13:37:55 smithi187 systemd[1]: Stopping Ceph mds.f for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:37:56.554 INFO:journalctl@ceph.mds.f.smithi187.stdout:Jan 24 13:37:56 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-f[81042]: 2024-01-24T13:37:56.172+0000 7f5a31b9c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T13:37:56.554 INFO:journalctl@ceph.mds.f.smithi187.stdout:Jan 24 13:37:56 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-f[81042]: 2024-01-24T13:37:56.172+0000 7f5a31b9c700 -1 mds.f *** got signal Terminated *** 2024-01-24T13:37:58.054 INFO:journalctl@ceph.mds.f.smithi187.stdout:Jan 24 13:37:57 smithi187 bash[84578]: ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-f 2024-01-24T13:37:58.110 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mds.f.service' 2024-01-24T13:37:58.144 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T13:37:58.145 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-01-24T13:37:58.145 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-01-24T13:37:58.145 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mds.i 2024-01-24T13:37:58.397 INFO:journalctl@ceph.mds.i.smithi187.stdout:Jan 24 13:37:58 smithi187 systemd[1]: Stopping Ceph mds.i for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:37:58.804 INFO:journalctl@ceph.mds.i.smithi187.stdout:Jan 24 13:37:58 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-i[81723]: 2024-01-24T13:37:58.396+0000 7f77432d7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T13:37:58.804 INFO:journalctl@ceph.mds.i.smithi187.stdout:Jan 24 13:37:58 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-i[81723]: 2024-01-24T13:37:58.396+0000 7f77432d7700 -1 mds.i *** got signal Terminated *** 2024-01-24T13:38:02.554 INFO:journalctl@ceph.mds.i.smithi187.stdout:Jan 24 13:38:02 smithi187 bash[84716]: ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-i 2024-01-24T13:38:02.639 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mds.i.service' 2024-01-24T13:38:02.673 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T13:38:02.673 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-01-24T13:38:02.674 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-01-24T13:38:02.674 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mds.l 2024-01-24T13:38:02.930 INFO:journalctl@ceph.mds.l.smithi187.stdout:Jan 24 13:38:02 smithi187 systemd[1]: Stopping Ceph mds.l for 83afcf5e-babc-11ee-95b1-87774f69a715... 2024-01-24T13:38:03.304 INFO:journalctl@ceph.mds.l.smithi187.stdout:Jan 24 13:38:02 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-l[80701]: 2024-01-24T13:38:02.929+0000 7f3f1c055700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T13:38:03.304 INFO:journalctl@ceph.mds.l.smithi187.stdout:Jan 24 13:38:02 smithi187 ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-l[80701]: 2024-01-24T13:38:02.929+0000 7f3f1c055700 -1 mds.l *** got signal Terminated *** 2024-01-24T13:38:05.305 INFO:journalctl@ceph.mds.l.smithi187.stdout:Jan 24 13:38:05 smithi187 bash[84854]: ceph-83afcf5e-babc-11ee-95b1-87774f69a715-mds-l 2024-01-24T13:38:05.358 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83afcf5e-babc-11ee-95b1-87774f69a715@mds.l.service' 2024-01-24T13:38:05.394 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T13:38:05.394 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-01-24T13:38:05.394 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 --force --keep-logs 2024-01-24T13:38:05.605 INFO:teuthology.orchestra.run.smithi029.stdout:Deleting cluster with fsid: 83afcf5e-babc-11ee-95b1-87774f69a715 2024-01-24T13:38:07.201 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 --force --keep-logs 2024-01-24T13:38:07.386 INFO:teuthology.orchestra.run.smithi163.stdout:Deleting cluster with fsid: 83afcf5e-babc-11ee-95b1-87774f69a715 2024-01-24T13:38:08.778 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 --force --keep-logs 2024-01-24T13:38:08.965 INFO:teuthology.orchestra.run.smithi187.stdout:Deleting cluster with fsid: 83afcf5e-babc-11ee-95b1-87774f69a715 2024-01-24T13:38:10.347 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-24T13:38:10.386 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-24T13:38:10.423 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-24T13:38:10.454 INFO:tasks.cephadm:Archiving crash dumps... 2024-01-24T13:38:10.456 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/crash to /home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530810/remote/smithi029/crash 2024-01-24T13:38:10.457 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/crash -- . 2024-01-24T13:38:10.503 INFO:teuthology.orchestra.run.smithi029.stderr:tar: /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/crash: Cannot open: No such file or directory 2024-01-24T13:38:10.504 INFO:teuthology.orchestra.run.smithi029.stderr:tar: Error is not recoverable: exiting now 2024-01-24T13:38:10.506 DEBUG:teuthology.misc:Transferring archived files from smithi163:/var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/crash to /home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530810/remote/smithi163/crash 2024-01-24T13:38:10.507 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar c -f - -C /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/crash -- . 2024-01-24T13:38:10.561 INFO:teuthology.orchestra.run.smithi163.stderr:tar: /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/crash: Cannot open: No such file or directory 2024-01-24T13:38:10.561 INFO:teuthology.orchestra.run.smithi163.stderr:tar: Error is not recoverable: exiting now 2024-01-24T13:38:10.563 DEBUG:teuthology.misc:Transferring archived files from smithi187:/var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/crash to /home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530810/remote/smithi187/crash 2024-01-24T13:38:10.564 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar c -f - -C /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/crash -- . 2024-01-24T13:38:10.603 INFO:teuthology.orchestra.run.smithi187.stderr:tar: /var/lib/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/crash: Cannot open: No such file or directory 2024-01-24T13:38:10.603 INFO:teuthology.orchestra.run.smithi187.stderr:tar: Error is not recoverable: exiting now 2024-01-24T13:38:10.604 INFO:tasks.cephadm:Checking cluster log for badness... 2024-01-24T13:38:10.604 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-01-24T13:38:10.645 INFO:teuthology.orchestra.run.smithi029.stderr:grep: /var/log/ceph/83afcf5e-babc-11ee-95b1-87774f69a715/ceph.log: No such file or directory 2024-01-24T13:38:10.648 INFO:tasks.cephadm:Compressing logs... 2024-01-24T13:38:10.648 DEBUG:teuthology.orchestra.run.smithi029:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-24T13:38:10.691 DEBUG:teuthology.orchestra.run.smithi163:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-24T13:38:10.693 DEBUG:teuthology.orchestra.run.smithi187:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-24T13:38:10.720 INFO:teuthology.orchestra.run.smithi187.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-01-24T13:38:10.723 INFO:teuthology.orchestra.run.smithi029.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-01-24T13:38:10.724 INFO:teuthology.orchestra.run.smithi163.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-01-24T13:38:11.980 INFO:tasks.cephadm:Archiving logs... 2024-01-24T13:38:11.980 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530810/remote/smithi029/log 2024-01-24T13:38:11.982 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-24T13:38:12.221 DEBUG:teuthology.misc:Transferring archived files from smithi163:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530810/remote/smithi163/log 2024-01-24T13:38:12.222 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-24T13:38:12.374 DEBUG:teuthology.misc:Transferring archived files from smithi187:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530810/remote/smithi187/log 2024-01-24T13:38:12.375 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-24T13:38:12.537 INFO:tasks.cephadm:Removing cluster... 2024-01-24T13:38:12.537 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 --force 2024-01-24T13:38:12.739 INFO:teuthology.orchestra.run.smithi029.stdout:Deleting cluster with fsid: 83afcf5e-babc-11ee-95b1-87774f69a715 2024-01-24T13:38:13.002 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 --force 2024-01-24T13:38:13.189 INFO:teuthology.orchestra.run.smithi163.stdout:Deleting cluster with fsid: 83afcf5e-babc-11ee-95b1-87774f69a715 2024-01-24T13:38:13.427 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 83afcf5e-babc-11ee-95b1-87774f69a715 --force 2024-01-24T13:38:13.613 INFO:teuthology.orchestra.run.smithi187.stdout:Deleting cluster with fsid: 83afcf5e-babc-11ee-95b1-87774f69a715 2024-01-24T13:38:13.852 INFO:tasks.cephadm:Removing cephadm ... 2024-01-24T13:38:13.852 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-24T13:38:13.875 DEBUG:teuthology.orchestra.run.smithi163:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-24T13:38:13.894 DEBUG:teuthology.orchestra.run.smithi187:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-24T13:38:13.912 INFO:tasks.cephadm:Teardown complete 2024-01-24T13:38:13.912 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_4e3182a65800aed6d4aa2f6b86da3cfd72188134/qa/tasks/cephadm.py", line 1714, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_4e3182a65800aed6d4aa2f6b86da3cfd72188134/qa/tasks/cephadm.py", line 1629, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_4e3182a65800aed6d4aa2f6b86da3cfd72188134/qa/tasks/cephadm.py", line 1051, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/git.ceph.com_ceph-c_4e3182a65800aed6d4aa2f6b86da3cfd72188134/qa/tasks/cephfs/filesystem.py", line 532, in __init__ self.create() File "/home/teuthworker/src/git.ceph.com_ceph-c_4e3182a65800aed6d4aa2f6b86da3cfd72188134/qa/tasks/cephfs/filesystem.py", line 762, in create self.run_ceph_cmd('fs', 'subvolumegroup', 'create', self.name, 'qa') AttributeError: 'Filesystem' object has no attribute 'run_ceph_cmd' 2024-01-24T13:38:14.129 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=1b898fa2d9c0437da20c40263e897529 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_4e3182a65800aed6d4aa2f6b86da3cfd72188134/qa/tasks/cephadm.py", line 1714, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_4e3182a65800aed6d4aa2f6b86da3cfd72188134/qa/tasks/cephadm.py", line 1629, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_4e3182a65800aed6d4aa2f6b86da3cfd72188134/qa/tasks/cephadm.py", line 1051, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/git.ceph.com_ceph-c_4e3182a65800aed6d4aa2f6b86da3cfd72188134/qa/tasks/cephfs/filesystem.py", line 532, in __init__ self.create() File "/home/teuthworker/src/git.ceph.com_ceph-c_4e3182a65800aed6d4aa2f6b86da3cfd72188134/qa/tasks/cephfs/filesystem.py", line 762, in create self.run_ceph_cmd('fs', 'subvolumegroup', 'create', self.name, 'qa') AttributeError: 'Filesystem' object has no attribute 'run_ceph_cmd' 2024-01-24T13:38:14.133 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-01-24T13:38:14.142 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-24T13:38:14.151 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-24T13:38:14.152 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-24T13:38:14.166 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-24T13:38:14.169 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-24T13:38:14.275 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-24T13:38:14.284 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-24T13:38:14.284 DEBUG:teuthology.orchestra.run.smithi029:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-24T13:38:14.287 DEBUG:teuthology.orchestra.run.smithi163:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-24T13:38:14.289 DEBUG:teuthology.orchestra.run.smithi187:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-24T13:38:14.303 INFO:teuthology.orchestra.run.smithi163.stderr:bash: ntpq: command not found 2024-01-24T13:38:14.305 INFO:teuthology.orchestra.run.smithi029.stderr:bash: ntpq: command not found 2024-01-24T13:38:14.306 INFO:teuthology.orchestra.run.smithi187.stderr:bash: ntpq: command not found 2024-01-24T13:38:14.309 INFO:teuthology.orchestra.run.smithi163.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-24T13:38:14.309 INFO:teuthology.orchestra.run.smithi163.stdout:=============================================================================== 2024-01-24T13:38:14.309 INFO:teuthology.orchestra.run.smithi163.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 12 -10us[ -10us] +/- 67ms 2024-01-24T13:38:14.309 INFO:teuthology.orchestra.run.smithi163.stdout:^* hv02.front.sepia.ceph.com 3 6 377 13 +247us[ +261us] +/- 67ms 2024-01-24T13:38:14.309 INFO:teuthology.orchestra.run.smithi163.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 13 -260us[ -246us] +/- 88ms 2024-01-24T13:38:14.309 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-24T13:38:14.311 INFO:teuthology.orchestra.run.smithi187.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-24T13:38:14.311 INFO:teuthology.orchestra.run.smithi187.stdout:=============================================================================== 2024-01-24T13:38:14.311 INFO:teuthology.orchestra.run.smithi187.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 12 -9967ns[+1477ns] +/- 67ms 2024-01-24T13:38:14.312 INFO:teuthology.orchestra.run.smithi187.stdout:^* hv02.front.sepia.ceph.com 3 6 377 11 +242us[ +254us] +/- 67ms 2024-01-24T13:38:14.312 INFO:teuthology.orchestra.run.smithi187.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 12 -264us[ -253us] +/- 88ms 2024-01-24T13:38:14.312 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-24T13:38:14.312 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-24T13:38:14.312 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2024-01-24T13:38:14.312 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 14 -12us[-1142ns] +/- 67ms 2024-01-24T13:38:14.312 INFO:teuthology.orchestra.run.smithi029.stdout:^* hv02.front.sepia.ceph.com 3 6 377 11 +231us[ +242us] +/- 67ms 2024-01-24T13:38:14.313 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 11 -275us[ -264us] +/- 88ms 2024-01-24T13:38:14.313 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-24T13:38:14.313 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-24T13:38:14.322 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-24T13:38:14.322 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-24T13:38:14.330 DEBUG:teuthology.orchestra.run.smithi029:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-24T13:38:14.435 DEBUG:teuthology.orchestra.run.smithi163:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-24T13:38:14.502 DEBUG:teuthology.orchestra.run.smithi187:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-24T13:38:14.570 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-24T13:38:14.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T13:38:14.609 DEBUG:teuthology.orchestra.run.smithi163:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-24T13:38:14.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T13:38:14.640 DEBUG:teuthology.orchestra.run.smithi187:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-24T13:38:14.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T13:38:14.671 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-24T13:38:14.681 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-24T13:38:14.689 INFO:teuthology.task.internal:Duration was 1373.067417 seconds 2024-01-24T13:38:14.689 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-24T13:38:14.698 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-24T13:38:14.698 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-24T13:38:14.701 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-24T13:38:14.703 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-24T13:38:14.748 INFO:teuthology.orchestra.run.smithi163.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-24T13:38:14.758 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-24T13:38:14.759 INFO:teuthology.orchestra.run.smithi187.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-24T13:38:15.115 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-24T13:38:15.115 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi029.front.sepia.ceph.com 2024-01-24T13:38:15.115 DEBUG:teuthology.orchestra.run.smithi029:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-24T13:38:15.143 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi163.front.sepia.ceph.com 2024-01-24T13:38:15.143 DEBUG:teuthology.orchestra.run.smithi163:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-24T13:38:15.166 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi187.front.sepia.ceph.com 2024-01-24T13:38:15.166 DEBUG:teuthology.orchestra.run.smithi187:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-24T13:38:15.189 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-24T13:38:15.190 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-24T13:38:15.192 DEBUG:teuthology.orchestra.run.smithi163:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-24T13:38:15.209 DEBUG:teuthology.orchestra.run.smithi187:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-24T13:38:15.280 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-24T13:38:15.281 DEBUG:teuthology.orchestra.run.smithi029:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-24T13:38:15.283 DEBUG:teuthology.orchestra.run.smithi163:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-24T13:38:15.286 DEBUG:teuthology.orchestra.run.smithi187:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-24T13:38:15.389 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-24T13:38:15.399 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-24T13:38:15.399 DEBUG:teuthology.orchestra.run.smithi029:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-24T13:38:15.430 DEBUG:teuthology.orchestra.run.smithi163:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-24T13:38:15.458 DEBUG:teuthology.orchestra.run.smithi187:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-24T13:38:15.486 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-24T13:38:15.495 DEBUG:teuthology.orchestra.run.smithi029:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-24T13:38:15.498 DEBUG:teuthology.orchestra.run.smithi163:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-24T13:38:15.501 DEBUG:teuthology.orchestra.run.smithi187:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-24T13:38:15.525 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern = core 2024-01-24T13:38:15.529 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = core 2024-01-24T13:38:15.552 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern = core 2024-01-24T13:38:15.586 DEBUG:teuthology.orchestra.run.smithi029:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-24T13:38:15.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T13:38:15.607 DEBUG:teuthology.orchestra.run.smithi163:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-24T13:38:15.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T13:38:15.630 DEBUG:teuthology.orchestra.run.smithi187:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-24T13:38:15.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T13:38:15.646 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-24T13:38:15.656 INFO:teuthology.task.internal:Transferring archived files... 2024-01-24T13:38:15.656 DEBUG:teuthology.misc:Transferring archived files from smithi029:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530810/remote/smithi029 2024-01-24T13:38:15.657 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-24T13:38:15.706 DEBUG:teuthology.misc:Transferring archived files from smithi163:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530810/remote/smithi163 2024-01-24T13:38:15.706 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-24T13:38:15.746 DEBUG:teuthology.misc:Transferring archived files from smithi187:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530810/remote/smithi187 2024-01-24T13:38:15.746 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-24T13:38:15.786 INFO:teuthology.task.internal:Removing archive directory... 2024-01-24T13:38:15.786 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-24T13:38:15.788 DEBUG:teuthology.orchestra.run.smithi163:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-24T13:38:15.791 DEBUG:teuthology.orchestra.run.smithi187:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-24T13:38:15.836 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-24T13:38:15.848 INFO:teuthology.task.internal:Not uploading archives. 2024-01-24T13:38:15.848 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-24T13:38:15.858 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-24T13:38:15.858 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-24T13:38:15.861 DEBUG:teuthology.orchestra.run.smithi163:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-24T13:38:15.863 DEBUG:teuthology.orchestra.run.smithi187:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-24T13:38:15.877 INFO:teuthology.orchestra.run.smithi163.stdout: 262157 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 24 13:38 /home/ubuntu/cephtest 2024-01-24T13:38:15.879 INFO:teuthology.orchestra.run.smithi029.stdout: 262175 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 24 13:38 /home/ubuntu/cephtest 2024-01-24T13:38:15.893 INFO:teuthology.orchestra.run.smithi187.stdout: 262157 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 24 13:38 /home/ubuntu/cephtest 2024-01-24T13:38:15.894 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-24T13:38:15.904 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-24T13:38:15.926 INFO:teuthology.nuke:Checking targets against current locks 2024-01-24T13:38:15.955 DEBUG:teuthology.nuke:shortname: smithi029 2024-01-24T13:38:15.955 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-24T13:38:15.980 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530810', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-24 13:09:09.630991', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCiK2VIXd6Nu30jJMgSMRgo2UZyHyGkunsspZGUXTboTnLdPBVmazxowZHf2sjurpFHT6+0u0dIu7055V2p51ig='} 2024-01-24T13:38:15.985 DEBUG:teuthology.nuke:shortname: smithi163 2024-01-24T13:38:15.985 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-24T13:38:16.013 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi163.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530810', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-24 13:09:09.629150', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGzEy4v5oV9PYT1rSTFgvNHNhlEnLzAgJW98x7PDsgydywKRd7tBCIF7SJdjd9v4Xsc7+BZF15W50BT3NTeJsKw='} 2024-01-24T13:38:16.014 DEBUG:teuthology.nuke:shortname: smithi187 2024-01-24T13:38:16.014 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-24T13:38:16.044 INFO:teuthology.orchestra.console.smithi029:Power off 2024-01-24T13:38:16.045 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-24T13:38:16.056 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi187.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530810', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-24 13:09:09.632676', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCXDV7vpbfuct6+Hq8JqpQryO10457dw7LMPhk4rLbFIUjKm0rpSLKVUdEk5m5bb4ZdqV+vw4Q3gaRW/IQuKL3o='} 2024-01-24T13:38:16.068 DEBUG:teuthology.orchestra.console.smithi029:power off output: Chassis Power Control: Down/Off 2024-01-24T13:38:16.069 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-24T13:38:16.088 INFO:teuthology.orchestra.console.smithi163:Power off 2024-01-24T13:38:16.089 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-24T13:38:16.104 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on 2024-01-24T13:38:16.112 DEBUG:teuthology.orchestra.console.smithi163:power off output: Chassis Power Control: Down/Off 2024-01-24T13:38:16.113 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-24T13:38:16.131 INFO:teuthology.orchestra.console.smithi187:Power off 2024-01-24T13:38:16.132 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-24T13:38:16.142 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is on 2024-01-24T13:38:16.154 DEBUG:teuthology.orchestra.console.smithi187:power off output: Chassis Power Control: Down/Off 2024-01-24T13:38:16.154 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-24T13:38:16.178 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is on 2024-01-24T13:38:20.105 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-24T13:38:20.143 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-24T13:38:20.179 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-24T13:38:20.216 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on 2024-01-24T13:38:20.253 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is on 2024-01-24T13:38:20.289 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is on 2024-01-24T13:38:24.220 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-24T13:38:24.253 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-24T13:38:24.291 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-24T13:38:24.330 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is off 2024-01-24T13:38:24.363 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is off 2024-01-24T13:38:24.401 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is on 2024-01-24T13:38:24.431 INFO:teuthology.orchestra.console.smithi029:Power off completed 2024-01-24T13:38:24.465 INFO:teuthology.orchestra.console.smithi163:Power off completed 2024-01-24T13:38:28.405 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-24T13:38:28.516 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is off 2024-01-24T13:38:28.616 INFO:teuthology.orchestra.console.smithi187:Power off completed 2024-01-24T13:38:28.723 INFO:teuthology.run:Summary data: description: fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}} duration: 1373.067417383194 failure_reason: '''Filesystem'' object has no attribute ''run_ceph_cmd''' flavor: default owner: scheduled_vshankar@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=1b898fa2d9c0437da20c40263e897529 status: fail success: false 2024-01-24T13:38:28.724 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-24T13:38:28.802 INFO:teuthology.run:FAIL